Managing inventory of perishable products

ABSTRACT

A system for identifying perishable products in a retail store based on analysis of image data and for automatically generating suggestions relating to the identified products is provided. The system may comprise at least one processor configured to: receive a set of images depicting a plurality of perishable products displayed on at least one shelving unit in a retail store; analyze the set of images to determine information about a displayed inventory of the plurality of perishable products; obtain information about additional perishable products scheduled to be displayed on the at least one shelving unit; use the information about the displayed inventory and the information about the additional perishable products to determine at least one suggestion regarding placement of perishable products in the retail store; and provide the at least one suggestion to an entity associated with the retail store.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of PCT International Application No.PCT/US2019/049528, filed Sep. 4, 2019, which claims the benefit ofpriority of U.S. Provisional Patent Application No. 62/727,301, filed onSep. 5, 2018, and U.S. Provisional Patent Application No. 62/773,427,filed on Nov. 30, 2018. All of the foregoing applications areincorporated herein by reference in their entirety.

BACKGROUND I. Technical Field

The present disclosure relates generally to systems, methods, anddevices for identifying products in retail stores, and more specificallyto systems, methods, and devices for capturing, collecting, andautomatically analyzing images of products displayed in retail storesfor purposes of providing one or more functions associated with theidentified products.

II. Background Information

Shopping in stores is a prevalent part of modern daily life. Storeowners (also known as “retailers”) stock a wide variety of products onstore shelves and add associated labels and promotions to the storeshelves. Typically, retailers have a set of processes and instructionsfor organizing products on the store shelves. The source of some ofthese instructions may include contractual obligations and otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. Nowadays, many retailers and supplierssend people to stores to personally monitor compliance with the desiredproduct placement. Such a monitoring technique, however, may beinefficient and may result in nonuniform compliance among retailersrelative to various product-related guidelines. This technique may alsoresult in significant gaps in compliance, as it does not allow forcontinuous monitoring of dynamically changing product displays. Toincrease productivity, among other potential benefits, there is atechnological need to provide a dynamic solution that will automaticallymonitor retail spaces. Such a solution, for example and among otherfeatures, may automatically determine whether a disparity exists betweena desired product placement and an actual product placement.

The disclosed devices and methods are directed to providing new ways formonitoring retail establishments using image processing and supportingsensors.

SUMMARY

Embodiments consistent with the present disclosure provide systems,methods, and devices for capturing, collecting, and analyzing images ofproducts displayed in retail stores. For example, consistent with thedisclosed embodiments, an example system may receive an image depictinga store shelf having products displayed thereon, identify the productson the store shelf, and trigger an alert when disparity exists betweenthe desired product placement and the actual product placement.

Consistent an embodiment, a system for processing images captured in aretail store and automatically identifying a product shortage isdisclosed. The system may include at least one processor configured toaccess information describing a desired placement of products of acertain product type on at least one store shelf. The at least oneprocessor may receive image data from a plurality of image sensorsmounted in the retail store, the image data including a plurality ofimages captured at different times and depicting the at least one storeshelf. The at least one processor may also analyze, from among theplurality of images, a first image captured at a first time to determinethat a first quantity of products of the certain product type at thefirst time exceeds a quantity threshold associated with the informationdescribing the desired placement of products. Additionally, the at leastone processor may analyze, from among the plurality of images, a secondimage captured at a second time later than the first time to determinethat a second quantity of products of the certain product type at thesecond time is below the quantity threshold associated with theinformation describing the desired placement of products. Based on thedetermination that the first quantity of products of the certain producttype at the first time exceeds the quantity threshold, the at least oneprocessor may avoid initiating an unscheduled action to cause restockingthe at least one store shelf with products of the certain product typeat least until a third time later than the second time. The at least oneprocessor may analyze, from among the plurality of images, a third imagecaptured at the third time to determine that a third quantity ofproducts of the certain product type at the third time is also below thequantity threshold associated with the information describing thedesired placement of products. Thereafter, based on the determinationthat the second quantity of products of the certain product type at thesecond time and the third quantity of products at the third time arebelow the quantity threshold, the at least one processor may provideinformation indicative of a product shortage at the at least one storeshelf.

Consistent with an embodiment, a method for processing images capturedin a retail store and automatically identifying a product shortage isdisclosed. The method may include accessing information describing adesired placement of products of a certain product type on at least onestore shelf. The method may include receiving image data from aplurality of image sensors mounted in a retail store, the image dataincluding a plurality of images captured at different times anddepicting the at least one store shelf. The method may also includeanalyzing, from among the plurality of images, a first image captured ata first time to determine that a first quantity of products of thecertain product type at the first time exceeds a quantity thresholdassociated with the information describing the desired placement ofproducts. The method may include analyzing, from among the plurality ofimages, a second image captured at a second time later than the firsttime to determine that a second quantity of products of the certainproduct type at the second time is below the quantity thresholdassociated with the information describing the desired placement ofproducts. Based on the determination that the first quantity of productsof the certain product type at the first time exceeds the quantitythreshold, the method may include avoiding initiating an unscheduledaction to cause restocking the at least one store shelf with products ofthe certain product type until a third time later than the second time.The method may include analyzing, from among the plurality of images, athird image captured at the third time to determine that a thirdquantity of products of the certain product type at the third time isalso below the quantity threshold associated with the informationdescribing the desired placement of products. Based on the determinationthat the second quantity of products of the certain product type at thesecond time and the third quantity of products at the third time arebelow the quantity threshold, the method may also include providinginformation indicative of a product shortage at the at least one storeshelf.

Consistent with an embodiment, a computer program product for processingimages captured in a retail store and automatically identifying aproduct shortage is disclosed. The computer program product may beembodied in a non-transitory computer-readable medium and may beexecutable by at least one processor. The computer program product mayinclude instructions for causing the at least one processor to execute amethod including accessing information describing a desired placement ofproducts of a certain product type on at least one store shelf. Themethod may include receiving image data from a plurality of imagesensors mounted in a retail store, the image data including a pluralityof images captured at different times and depicting the at least onestore shelf. The method may also include analyzing, from among theplurality of images, a first image captured at a first time to determinethat a first quantity of products of the certain product type at thefirst time exceeds a quantity threshold associated with the informationdescribing the desired placement of products. The method may includeanalyzing, from among the plurality of images, a second image capturedat a second time later than the first time to determine that a secondquantity of products of the certain product type at the second time isbelow the quantity threshold associated with the information describingthe desired placement of products. Based on the determination that thefirst quantity of products of the certain product type at the first timeexceeds the quantity threshold, the method may include avoidinginitiating an unscheduled action to cause restocking the at least onestore shelf with products of the certain product type until a third timelater than the second time. The method may also include analyzing, fromamong the plurality of images, a third image captured at the third timeto determine that a third quantity of products of the certain producttype at the third time is also below the quantity threshold associatedwith the information describing the desired placement of products. Basedon the determination that the second quantity of products of the certainproduct type at the second time and the third quantity of products atthe third time are below the quantity threshold, the method may includeproviding information indicative of a product shortage at the at leastone store shelf.

Consistent with an embodiment, a system for reducing product shortagedurations in retail stores based on analysis of image data is disclosed.The system may include a communication interface configured to receiveimage data from a plurality of retail stores, wherein the image data isindicative of a product shortage of at least one product type relativeto information describing a desired placement of products of the atleast one product type on at least one store shelf. The system may alsoinclude at least one processor configured to analyze the image data todetect a plurality of occurrences of product shortages of the at leastone product type in the plurality of retail stores and determine productshortage durations associated with the plurality of occurrences. The atleast one processor may identify at least one common factor contributingto product shortage duration of at least part of the plurality ofoccurrences of the product shortages of the at least one product type inthe plurality of retail stores. The at least one processor may alsodetermine an action, associated with the at least one common factor, forpotentially reducing product shortage durations of future shortages ofthe at least one product type in the plurality of retail stores.Additionally, the at least one processor may provide informationassociated with the identified action to an entity.

Consistent with an embodiment, a method reducing product shortagedurations in retail stores based on analysis of image data is disclosed.The method may include receiving image data from a plurality of retailstores, wherein the image data is indicative of a product shortage of atleast one product type relative to information describing a desiredplacement of products of the at least one product type on at least onestore shelf. The method may include analyzing the image data to detect aplurality of occurrences of product shortages of the at least oneproduct type in the plurality of retail stores and determine productshortage durations for the plurality of occurrences. The method may alsoinclude identifying at least one common factor contributing to productshortage duration of at least part of the plurality of occurrences ofthe product shortages of the at least one product type in the pluralityof retail stores. The method may include determining an action,associated with the at least one common factor, for potentially reducingproduct shortage durations of future shortages of the at least oneproduct type in the plurality of retail stores. Finally, the method mayinclude providing information associated with the identified action toan entity.

Consistent with an embodiment, a computer program product for reducingproduct shortage durations in retail stores based on analysis of imagedata is disclosed. The computer program product may be embodied in anon-transitory computer-readable medium and may be executable by atleast one processor. The computer program product may includeinstructions for causing the at least one processor to execute a method.The method may include receiving image data from a plurality of retailstores, wherein the image data is indicative of a product shortage of atleast one product type relative to information describing a desiredplacement of products of the at least one product type on at least onestore shelf. The method may include analyzing the image data to detect aplurality of occurrences of product shortages of the at least oneproduct type in the plurality of retail stores and determine productshortage durations for the plurality of occurrences. The method may alsoinclude identifying at least one common factor contributing to productshortage duration of at least part of the plurality of occurrences ofthe product shortages of the at least one product type in the pluralityof retail stores. The method may include determining an action,associated with the at least one common factor, for potentially reducingproduct shortage durations of future shortages of the at least oneproduct type in the plurality of retail stores. Finally, the method mayinclude providing information associated with the identified action toan entity.

Consistent with an embodiment, a system for identifying perishableproducts in a retail store based on analysis of image data and forautomatically generating suggestions relating to the identified productsis disclosed. The system may include at least one processor configuredto receive a set of images depicting a plurality of perishable productsdisplayed on at least one shelving unit in a retail store. The at leastone processor may analyze the set of images to determine informationabout a displayed inventory of the plurality of perishable products. Theat least one processor may also obtain information about additionalperishable products scheduled to be displayed on the at least oneshelving unit. Additionally, the at least one processor may use theinformation about the displayed inventory and the information about theadditional perishable products to determine at least one suggestionregarding placement of perishable products in the retail store.Thereafter, the at least one processor may provide the at least onesuggestion to an entity associated with the retail store.

Consistent with an embodiment, a system for managing inventory ofperishable products in retail stores is disclosed. The system mayinclude at least one processor configured to receive a set of imagesdepicting a plurality of perishable products displayed on at least oneshelving unit in a retail store. The at least one processor may analyzethe set of images to determine whether a quantity of the products of acertain type of perishable products is below a quantity threshold. Inresponse to a determination that the quantity of the products of thecertain type of perishable products is below the quantity threshold, theat least one processor may initiate an action for restocking theshelving unit with additional products from the certain type ofperishable products. And in response to a determination that thequantity of the products of the certain type of perishable products isabove the quantity threshold, the at least one processor may forgoinitiating the action for restocking the shelving unit with additionalproducts from the certain type of perishable products. The at least oneprocessor may also analyze the set of images to determine that thequantity of the products of the certain type of perishable products isabove the quantity threshold, but a quality associated with the productsof the certain type of perishable products is below a quality threshold.In response to a determination that the quantity of the products of thecertain type of perishable products is above the quantity threshold butthe quality associated with the products of the certain type ofperishable products is below the quality threshold, the at least oneprocessor may initiate an action for improving the quality associatedwith the products of the certain type of perishable products.

Consistent with an embodiment, a system for managing inventory ofperishable products in retail stores is disclosed. The system mayinclude at least one processor configured to receive a set of imagesdepicting a plurality of perishable products displayed on at least oneshelving unit in a retail store. The at least one processor may identifyin the set of images representations of one or more products of a firsttype of perishable products and one or more products of a second type ofperishable products different from the first type of perishable product.The at least one processor may also analyze the set of images todetermine information about a displayed inventory of the first type ofperishable product and to determine information about a displayedinventory of the second type of perishable product. Based on thedetermined information about the displayed inventory of the first typeof perishable product, the at least one processor may initiate an actionprompting a clearance sale of the products of the first type ofperishable products. Furthermore, based on the determined informationabout the displayed inventory of the first type of perishable productand the determined information about the displayed inventory of thesecond type of perishable products, the at least one processor may avoidinitiating an action prompting a clearance sale of products of thesecond type of perishable products.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is an illustration of an exemplary system for analyzinginformation collected from a retail store;

FIG. 2 is a block diagram that illustrates some of the components of animage processing system, consistent with the present disclosure;

FIG. 3 is a block diagram that illustrates an exemplary embodiment of acapturing device, consistent with the present disclosure;

FIG. 4A is a schematic illustration of an example configuration forcapturing image data in a retail store, consistent with the presentdisclosure;

FIG. 4B is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure;

FIG. 4C is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure;

FIG. 5A is an illustration of an example system for acquiring images ofproducts in a retail store, consistent with the present disclosure.

FIG. 5B is an illustration of a shelf-mounted camera unit included in afirst housing of the example system of FIG. 5A, consistent with thepresent disclosure.

FIG. 5C is an exploded view illustration of a processing unit includedin a second housing of the example system of FIG. 5A, consistent withthe present disclosure.

FIG. 6A is a top view representation of an aisle in a retail store withmultiple image acquisition systems deployed thereon for acquiring imagesof products, consistent with the present disclosure.

FIG. 6B is a perspective view representation of part of a retailshelving unit with multiple image acquisition systems deployed thereonfor acquiring images of products, consistent with the presentdisclosure.

FIG. 6C provides a diagrammatic representation of how the exemplarydisclosed image acquisition systems may be positioned relative to retailshelving to acquire product images, consistent with the presentdisclosure.

FIG. 7A provides a flowchart of an exemplary method for acquiring imagesof products in retail store, consistent with the present disclosure.

FIG. 7B provides a flowchart of a method for acquiring images ofproducts in retail store, consistent with the present disclosure.

FIG. 8A is a schematic illustration of an example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure;

FIG. 8B is a schematic illustration of another example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure;

FIG. 9 is a schematic illustration of example configurations fordetection elements on store shelves, consistent with the presentdisclosure;

FIG. 10A illustrates an exemplary method for monitoring planogramcompliance on a store shelf, consistent with the present disclosure;

FIG. 10B is illustrates an exemplary method for triggering imageacquisition based on product events on a store shelf, consistent withthe present disclosure;

FIG. 11A is a schematic illustration of an example output for a marketresearch entity associated with the retail store, consistent with thepresent disclosure;

FIG. 11B is a schematic illustration of an example output for a supplierof the retail store, consistent with the present disclosure;

FIG. 11C is a schematic illustration of an example output for a managerof the retail store, consistent with the present disclosure;

FIG. 11D is a schematic illustration of two examples outputs for anemployee of the retail store, consistent with the present disclosure;and

FIG. 11E is a schematic illustration of an example output for an onlinecustomer of the retail store, consistent with the present disclosure.

FIG. 12 is a block diagram illustrating an exemplary embodiment of amemory device containing software modules for executing methodsconsistent with the present disclosure.

FIG. 13A is an illustration of a shelving unit at a first timeconsistent with the present disclosure.

FIG. 13B is an illustration of a shelving unit at a second timeconsistent with the present disclosure.

FIG. 13C is an illustration of a shelving unit at a third timeconsistent with the present disclosure.

FIG. 14A is a flowchart of an exemplary method for processing imagescaptured in a retail store and automatically identifying a productshortage consistent with the present disclosure.

FIG. 14B is a flowchart of an exemplary process for automaticallyidentifying a product shortage consistent with the present disclosure.

FIG. 14C is a flowchart of another exemplary process for automaticallyidentifying a product shortage consistent with the present disclosure.

FIG. 15 is a block diagram illustrating an exemplary embodiment of amemory device containing software modules for executing methodsconsistent with the present disclosure.

FIG. 16 is a flowchart of an exemplary process for reducing productshortage durations in retail stores based on analysis of image dataconsistent with the present disclosure.

FIG. 17A is a schematic illustration of an example output for a retailstore consistent with the present disclosure.

FIG. 17B is a schematic illustration of an example output for a marketresearch entity associated with a retail store consistent with thepresent disclosure.

FIG. 17C is a schematic illustration of an example output for a supplierassociated with the retail store consistent with the present disclosure.

FIG. 18 is a block diagram illustrating an exemplary embodiment of amemory device containing software modules for executing methodsconsistent with the present disclosure.

FIG. 19 is a diagram illustrating example outputs of the softwaremodules depicted in FIG. 18 consistent with the present disclosure.

FIG. 20 is a diagram illustrating an example of how an inventory ofperishable products in retail stores is managed based on image data andscheduling data consistent with the present disclosure.

FIG. 21 is a flowchart of an exemplary method consistent with anembodiment for managing inventory of perishable products in retailstores consistent with the present disclosure.

FIG. 22 is a graph showing example changes in quality and quantity ofperishable products from a certain type in a retail store consistentwith the present disclosure.

FIG. 23 is a flowchart of an exemplary process consistent with anembodiment for managing inventory of perishable products in retailstores consistent with the present disclosure.

FIG. 24 is an illustration of a shelving unit with display areas for twodifferent types of perishable products, consistent with the presentdisclosure.

FIG. 25 includes three bar diagrams illustrating an implementation of anembodiment for managing inventory of perishable products in retailstores consistent with the present disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

The present disclosure is directed to systems and methods for processingimages captured in a retail store. As used herein, the term “retailstore” or simply “store” refers to an establishment offering productsfor sale by direct selection by customers physically or virtuallyshopping within the establishment. The retail store may be anestablishment operated by a single retailer (e.g., supermarket) or anestablishment that includes stores operated by multiple retailers (e.g.,a shopping mall). Embodiments of the present disclosure includereceiving an image depicting a store shelf having at least one productdisplayed thereon. As used herein, the term “store shelf” or simply“shelf” refers to any suitable physical structure which may be used fordisplaying products in a retail environment. In one embodiment the storeshelf may be part of a shelving unit including a number of individualstore shelves. In another embodiment, the store shelf may include adisplay unit having a single-level or multi-level surfaces.

Consistent with the present disclosure, the system may process imagesand image data acquired by a capturing device to determine informationassociated with products displayed in the retail store. The term“capturing device” refers to any device configured to acquire image datarepresentative of products displayed in the retail store. Examples ofcapturing devices may include a digital camera, a time-of-flight camera,a stereo camera, an active stereo camera, a depth camera, a Lidarsystem, a laser scanner, CCD based devices, or any other sensor basedsystem capable of converting received light into electric signals. Theterm “image data” refers to any form of data generated based on opticalsignals in the near-infrared, infrared, visible, and ultravioletspectrums (or any other suitable radiation frequency range). Consistentwith the present disclosure, the image data may include pixel datastreams, digital images, digital video streams, data derived fromcaptured images, and data that may be used to construct a 3D image. Theimage data acquired by a capturing device may be transmitted by wired orwireless transmission to a remote server. In one embodiment, thecapturing device may include a stationary camera with communicationlayers (e.g., a dedicated camera fixed to a store shelf, a securitycamera, etc.). Such an embodiment is described in greater detail belowwith reference to FIG. 4A. In another embodiment, the capturing devicemay include a handheld device (e.g., a smartphone, a tablet, a mobilestation, a personal digital assistant, a laptop, and more) or a wearabledevice (e.g., smart glasses, a smartwatch, a clip-on camera). Such anembodiment is described in greater detail below with reference to FIG.4B. In another embodiment, the capturing device may include a roboticdevice with one or more cameras operated remotely or autonomously (e.g.,an autonomous robotic device, a drone, a robot on a track, and more).Such an embodiment is described in greater detail below with referenceto FIG. 4C.

In some embodiments, the capturing device may include one or more imagesensors. The term “image sensor” refers to a device capable of detectingand converting optical signals in the near-infrared, infrared, visible,and ultraviolet spectrums into electrical signals. The electricalsignals may be used to form image data (e.g., an image or a videostream) based on the detected signal. Examples of image sensors mayinclude semiconductor charge-coupled devices (CCD), active pixel sensorsin complementary metal-oxide-semiconductor (CMOS), or N-typemetal-oxide-semiconductors (NMOS, Live MOS). In some cases, the imagesensor may be part of a camera included in the capturing device.

Embodiments of the present disclosure further include analyzing imagesto detect and identify different products. As used herein, the term“detecting a product” may broadly refer to determining an existence ofthe product. For example, the system may determine the existence of aplurality of distinct products displayed on a store shelf. By detectingthe plurality of products, the system may acquire different detailsrelative to the plurality of products (e.g., how many products on astore shelf are associated with a same product type), but it does notnecessarily gain knowledge of the type of product. In contrast, the term“identifying a product” may refer to determining a unique identifierassociated with a specific type of product that allows inventorymanagers to uniquely refer to each product type in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificbrand of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific brand in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificcategory of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific category in a product catalogue. Insome embodiments, the identification may be made based at least in parton visual characteristics of the product (e.g., size, shape, logo, text,color, etc.). The unique identifier may include any codes that may beused to search a catalog, such as a series of digits, letters, symbols,or any combinations of digits, letters, and symbols. Consistent with thepresent disclosure, the terms “determining a type of a product” and“determining a product type” may also be used interchangeably in thisdisclosure with reference to the term “identifying a product.”

Embodiments of the present disclosure further include determining atleast one characteristic of the product for determining the type of theproduct. As used herein, the term “characteristic of the product” refersto one or more visually discernable features attributed to the product.Consistent with the present disclosure, the characteristic of theproduct may assist in classifying and identifying the product. Forexample, the characteristic of the product may be associated with theornamental design of the product, the size of the product, the shape ofthe product, the colors of the product, the brand of the product, a logoor text associated with the product (e.g., on a product label), andmore. In addition, embodiments of the present disclosure further includedetermining a confidence level associated with the determined type ofthe product. The term “confidence level” refers to any indication,numeric or otherwise, of a level (e.g., within a predetermined range)indicative of an amount of confidence the system has that the determinedtype of the product is the actual type of the product. For example, theconfidence level may have a value between 1 and 10, alternatively, theconfidence level may be expressed as a percentage.

In some cases, the system may compare the confidence level to athreshold. The term “threshold” as used herein denotes a referencevalue, a level, a point, or a range of values, for which, when theconfidence level is above it (or below it depending on a particular usecase), the system may follow a first course of action and, when theconfidence level is below it (or above it depending on a particular usecase), the system may follow a second course of action. The value of thethreshold may be predetermined for each type of product or may bedynamically selected based on different considerations. In oneembodiment, when the confidence level associated with a certain productis below a threshold, the system may obtain contextual information toincrease the confidence level. As used herein, the term “contextualinformation” (or “context”) refers to any information having a direct orindirect relationship with a product displayed on a store shelf. In someembodiments, the system may retrieve different types of contextualinformation from captured image data and/or from other data sources. Insome cases, contextual information may include recognized types ofproducts adjacent to the product under examination. In other cases,contextual information may include text appearing on the product,especially where that text may be recognized (e.g., via OCR) andassociated with a particular meaning. Other examples of types ofcontextual information may include logos appearing on the product, alocation of the product in the retail store, a brand name of theproduct, a price of the product, product information collected frommultiple retail stores, product information retrieved from a catalogassociated with a retail store, etc.

Reference is now made to FIG. 1, which shows an example of a system 100for analyzing information collected from retail stores 105 (for example,retail store 105A, retail store 105B, and retail store 105C). In oneembodiment, system 100 may represent a computer-based system that mayinclude computer system components, desktop computers, workstations,tablets, handheld computing devices, memory devices, and/or internalnetwork(s) connecting the components. System 100 may include or beconnected to various network computing resources (e.g., servers,routers, switches, network connections, storage devices, etc.) necessaryto support the services provided by system 100. In one embodiment,system 100 may enable identification of products in retail stores 105based on analysis of captured images. In another embodiment, system 100may enable a supply of information based on analysis of captured imagesto a market research entity 110 and to different suppliers 115 of theidentified products in retail stores 105 (for example, supplier 115A,supplier 115B, and supplier 115C). In another embodiment, system 100 maycommunicate with a user 120 (sometimes referred to herein as a customer,but which may include individuals associated with a retail environmentother than customers, such as store employee, data collection agent,etc.) about different products in retail stores 105. In one example,system 100 may receive images of products captured by user 120. Inanother example, system 100 may provide to user 120 informationdetermined based on automatic machine analysis of images captured by oneor more capturing devices 125 associated with retail stores 105.

System 100 may also include an image processing unit 130 to execute theanalysis of images captured by the one or more capturing devices 125.Image processing unit 130 may include a server 135 operatively connectedto a database 140. Image processing unit 130 may include one or moreservers connected by a communication network, a cloud platform, and soforth. Consistent with the present disclosure, image processing unit 130may receive raw or processed data from capturing device 125 viarespective communication links, and provide information to differentsystem components using a network 150. Specifically, image processingunit 130 may use any suitable image analysis technique including, forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc. In addition, image processing unit 130 may useclassification algorithms to distinguish between the different productsin the retail store. In some embodiments, image processing unit 130 mayutilize suitably trained machine learning algorithms and models toperform the product identification. Network 150 may facilitatecommunications and data exchange between different system componentswhen these components are coupled to network 150 to enable output ofdata derived from the images captured by the one or more capturingdevices 125. In some examples, the types of outputs that imageprocessing unit 130 can generate may include identification of products,indicators of product quantity, indicators of planogram compliance,indicators of service-improvement events (e.g., a cleaning event, arestocking event, a rearrangement event, etc.), and various reportsindicative of the performances of retail stores 105. Additional examplesof the different outputs enabled by image processing unit 130 aredescribed below with reference to FIGS. 11A-11E and throughout thedisclosure.

Consistent with the present disclosure, network 150 may be any type ofnetwork (including infrastructure) that provides communications,exchanges information, and/or facilitates the exchange of informationbetween the components of system 100. For example, network 150 mayinclude or be part of the Internet, a Local Area Network, wirelessnetwork (e.g., a Wi-Fi/302.11 network), or other suitable connections.In other embodiments, one or more components of system 100 maycommunicate directly through dedicated communication links, such as, forexample, a telephone network, an extranet, an intranet, the Internet,satellite communications, off-line communications, wirelesscommunications, transponder communications, a local area network (LAN),a wide area network (WAN), a virtual private network (VPN), and soforth.

In one example configuration, server 135 may be a cloud server thatprocesses images received directly (or indirectly) from one or morecapturing device 125 and processes the images to detect and/or identifyat least some of the plurality of products in the image based on visualcharacteristics of the plurality of products. The term “cloud server”refers to a computer platform that provides services via a network, suchas the Internet. In this example configuration, server 135 may usevirtual machines that may not correspond to individual hardware. Forexample, computational and/or storage capabilities may be implemented byallocating appropriate portions of desirable computation/storage powerfrom a scalable repository, such as a data center or a distributedcomputing environment. In one example, server 135 may implement themethods described herein using customized hard-wired logic, one or moreApplication Specific Integrated Circuits (ASICs) or Field ProgrammableGate Arrays (FPGAs), firmware, and/or program logic which, incombination with the computer system, cause server 135 to be aspecial-purpose machine.

In another example configuration, server 135 may be part of a systemassociated with a retail store that communicates with capturing device125 using a wireless local area network (WLAN) and may provide similarfunctionality as a cloud server. In this example configuration, server135 may communicate with an associated cloud server (not shown) andcloud database (not shown). The communications between the store serverand the cloud server may be used in a quality enforcement process, forupgrading the recognition engine and the software from time to time, forextracting information from the store level to other data users, and soforth. Consistent with another embodiment, the communications betweenthe store server and the cloud server may be discontinuous (purposely orunintentional) and the store server may be configured to operateindependently from the cloud server. For example, the store server maybe configured to generate a record indicative of changes in productplacement that occurred when there was a limited connection (or noconnection) between the store server and the cloud server, and toforward the record to the cloud server once connection is reestablished.

As depicted in FIG. 1, server 135 may be coupled to one or more physicalor virtual storage devices such as database 140. Server 135 may accessdatabase 140 to detect and/or identify products. The detection may occurthrough analysis of features in the image using an algorithm and storeddata. The identification may occur through analysis of product featuresin the image according to stored product models. Consistent with thepresent embodiment, the term “product model” refers to any type ofalgorithm or stored product data that a processor may access or executeto enable the identification of a particular product associated with theproduct model. For example, the product model may include a descriptionof visual and contextual properties of the particular product (e.g., theshape, the size, the colors, the texture, the brand name, the price, thelogo, text appearing on the particular product, the shelf associatedwith the particular product, adjacent products in a planogram, thelocation within the retail store, etc.). In some embodiments, a singleproduct model may be used by server 135 to identify more than one typeof products, such as, when two or more product models are used incombination to enable identification of a product. For example, in somecases, a first product model may be used by server 135 to identify aproduct category (such models may apply to multiple product types, e.g.,shampoo, soft drinks, etc.), and a second product model may be used byserver 135 to identify the product type, product identity, or othercharacteristics associated with a product. In some cases, such productmodels may be applied together (e.g., in series, in parallel, in acascade fashion, in a decision tree fashion, etc.) to reach a productidentification. In other embodiments, a single product model may be usedby server 135 to identify a particular product type (e.g., 6-pack of 16oz Coca-Cola Zero).

Database 140 may be included on a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other type ofstorage device or tangible or non-transitory computer-readable medium.Database 140 may also be part of server 135 or separate from server 135.When database 140 is not part of server 135, server 135 may exchangedata with database 140 via a communication link. Database 140 mayinclude one or more memory devices that store data and instructions usedto perform one or more features of the disclosed embodiments. In oneembodiment, database 140 may include any suitable databases, rangingfrom small databases hosted on a work station to large databasesdistributed among data centers. Database 140 may also include anycombination of one or more databases controlled by memory controllerdevices (e.g., server(s), etc.) or software. For example, database 140may include document management systems, Microsoft SQL databases,SharePoint databases, Oracle™ databases, Sybase™ databases, otherrelational databases, or non-relational databases, such as mongo andothers.

Consistent with the present disclosure, image processing unit 130 maycommunicate with output devices 145 to present information derived basedon processing of image data acquired by capturing devices 125. The term“output device” is intended to include all possible types of devicescapable of outputting information from server 135 to users or othercomputer systems (e.g., a display screen, a speaker, a desktop computer,a laptop computer, mobile device, tablet, a PDA, etc.), such as 145A,145B, 145C and 145D. In one embodiment each of the different systemcomponents (i.e., retail stores 105, market research entity 110,suppliers 115, and users 120) may be associated with an output device145, and each system component may be configured to present differentinformation on the output device 145. In one example, server 135 mayanalyze acquired images including representations of shelf spaces. Basedon this analysis, server 135 may compare shelf spaces associated withdifferent products, and output device 145A may present market researchentity 110 with information about the shelf spaces associated withdifferent products. The shelf spaces may also be compared with salesdata, expired products data, and more. Consistent with the presentdisclosure, market research entity 110 may be a part of (or may workwith) supplier 115. In another example, server 135 may determine productcompliance to a predetermined planogram, and output device 145B maypresent to supplier 115 information about the level of productcompliance at one or more retail stores 105 (for example in a specificretail store 105, in a group of retail stores 105 associated withsupplier 115, in all retail stores 105, and so forth). The predeterminedplanogram may be associated with contractual obligations and/or otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. In another example, server 135 maydetermine that a specific store shelf has a type of fault in the productplacement, and output device 145C may present to a manager of retailstore 105 a user-notification that may include information about acorrect display location of a misplaced product, information about astore shelf associated with the misplaced product, information about atype of the misplaced product, and/or a visual depiction of themisplaced product. In another example, server 135 may identify whichproducts are available on the shelf and output device 145D may presentto user 120 an updated list of products.

The components and arrangements shown in FIG. 1 are not intended tolimit the disclosed embodiments, as the system components used toimplement the disclosed processes and features may vary. In oneembodiment, system 100 may include multiple servers 135, and each server135 may host a certain type of service. For example, a first server mayprocess images received from capturing devices 125 to identify at leastsome of the plurality of products in the image, and a second server maydetermine from the identified products in retail stores 105 compliancewith contractual obligations between retail stores 105 and suppliers115. In another embodiment, system 100 may include multiple servers 135,a first type of servers 135 that may process information from specificcapturing devices 125 (e.g., handheld devices of data collection agents)or from specific retail stores 105 (e.g., a server dedicated to aspecific retail store 105 may be placed in or near the store). System100 may further include a second type of servers 135 that collect andprocess information from the first type of servers 135.

FIG. 2 is a block diagram representative of an example configuration ofserver 135. In one embodiment, server 135 may include a bus 200 (or anyother communication mechanism) that interconnects subsystems andcomponents for transferring information within server 135. For example,bus 200 may interconnect a processing device 202, a memory interface204, a network interface 206, and a peripherals interface 208 connectedto an I/O system 210.

Processing device 202, shown in FIG. 2, may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to execute particular instructionsassociated with embodiments described in the present disclosure. Theterm “processing device” refers to any physical device having anelectric circuit that performs a logic operation. For example,processing device 202 may include one or more processors, integratedcircuits, microchips, microcontrollers, microprocessors, all or part ofa central processing unit (CPU), graphics processing unit (GPU), digitalsignal processor (DSP), field programmable gate array (FPGA), or othercircuits suitable for executing instructions or performing logicoperations. Processing device 202 may include at least one processorconfigured to perform functions of the disclosed methods such as amicroprocessor manufactured by Intel™, Nvidia™, manufactured by AMD™,and so forth. Processing device 202 may include a single core ormultiple core processors executing parallel processes simultaneously. Inone example, processing device 202 may be a single core processorconfigured with virtual processing technologies. Processing device 202may implement virtual machine technologies or other technologies toprovide the ability to execute, control, run, manipulate, store, etc.,multiple software processes, applications, programs, etc. In anotherexample, processing device 202 may include a multiple-core processorarrangement (e.g., dual, quad core, etc.) configured to provide parallelprocessing functionalities to allow a device associated with processingdevice 202 to execute multiple processes simultaneously. It isappreciated that other types of processor arrangements could beimplemented to provide the capabilities disclosed herein.

Consistent with the present disclosure, the methods and processesdisclosed herein may be performed by server 135 as a result ofprocessing device 202 executing one or more sequences of one or moreinstructions contained in a non-transitory computer-readable storagemedium. As used herein, a non-transitory computer-readable storagemedium refers to any type of physical memory on which information ordata readable by at least one processor can be stored. Examples includerandom access memory (RAM), read-only memory (ROM), volatile memory,nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, anyother optical data storage medium, any physical medium with patterns ofholes, a RAM, a PROM, an EPROM, a FLASH-EPROM or any other flash memory,NVRAM, a cache, a register, any other memory chip or cartridge, andnetworked versions of the same. The terms “memory” and“computer-readable storage medium” may refer to multiple structures,such as a plurality of memories or computer-readable storage mediumslocated within server 135, or at a remote location. Additionally, one ormore computer-readable storage mediums can be utilized in implementing acomputer-implemented method. The term “computer-readable storage medium”should be understood to include tangible items and exclude carrier wavesand transient signals.

According to one embodiment, server 135 may include network interface206 (which may also be any communications interface) coupled to bus 200.Network interface 206 may provide one-way or two-way data communicationto a local network, such as network 150. Network interface 206 mayinclude an integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example, networkinterface 206 may include a local area network (LAN) card to provide adata communication connection to a compatible LAN. In anotherembodiment, network interface 206 may include an Ethernet port connectedto radio frequency receivers and transmitters and/or optical (e.g.,infrared) receivers and transmitters. The specific design andimplementation of network interface 206 depends on the communicationsnetwork(s) over which server 135 is intended to operate. As describedabove, server 135 may be a cloud server or a local server associatedwith retail store 105. In any such implementation, network interface 206may be configured to send and receive electrical, electromagnetic, oroptical signals, through wires or wirelessly, that may carry analog ordigital data streams representing various types of information. Inanother example, the implementation of network interface 206 may besimilar or identical to the implementation described below for networkinterface 306.

Server 135 may also include peripherals interface 208 coupled to bus200. Peripherals interface 208 may be connected to sensors, devices, andsubsystems to facilitate multiple functionalities. In one embodiment,peripherals interface 208 may be connected to I/O system 210 configuredto receive signals or input from devices and provide signals or outputto one or more devices that allow data to be received and/or transmittedby server 135. In one embodiment I/O system 210 may include or beassociated with output device 145. For example, I/O system 210 mayinclude a touch screen controller 212, an audio controller 214, and/orother input controller(s) 216. Touch screen controller 212 may becoupled to a touch screen 218. Touch screen 218 and touch screencontroller 212 can, for example, detect contact, movement, or breakthereof using any of a plurality of touch sensitivity technologies,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies as well as other proximity sensorarrays or other elements for determining one or more points of contactwith touch screen 218. Touch screen 218 may also, for example, be usedto implement virtual or soft buttons and/or a keyboard. In addition toor instead of touch screen 218, I/O system 210 may include a displayscreen (e.g., CRT, LCD, etc.), virtual reality device, augmented realitydevice, and so forth. Specifically, touch screen controller 212 (ordisplay screen controller) and touch screen 218 (or any of thealternatives mentioned above) may facilitate visual output from server135. Audio controller 214 may be coupled to a microphone 220 and aspeaker 222 to facilitate voice-enabled functions, such as voicerecognition, voice replication, digital recording, and telephonyfunctions. Specifically, audio controller 214 and speaker 222 mayfacilitate audio output from server 135. The other input controller(s)216 may be coupled to other input/control devices 224, such as one ormore buttons, keyboards, rocker switches, thumb-wheel, infrared port,USB port, image sensors, motion sensors, depth sensors, and/or a pointerdevice such as a computer mouse or a stylus.

In some embodiments, processing device 202 may use memory interface 204to access data and a software product stored on a memory device 226.Memory device 226 may include operating system programs for server 135that perform operating system functions when executed by the processingdevice. By way of example, the operating system programs may includeMicrosoft Windows™, Unix™ Linux™, Apple™ operating systems, personaldigital assistant (PDA) type operating systems such as Apple iOS, GoogleAndroid, Blackberry OS, or other types of operating systems.

Memory device 226 may also store communication instructions 228 tofacilitate communicating with one or more additional devices (e.g.,capturing device 125), one or more computers (e.g., output devices145A-145D) and/or one or more servers. Memory device 226 may includegraphical user interface instructions 230 to facilitate graphic userinterface processing; image processing instructions 232 to facilitateimage data processing-related processes and functions; sensor processinginstructions 234 to facilitate sensor-related processing and functions;web browsing instructions 236 to facilitate web browsing-relatedprocesses and functions; and other software instructions 238 tofacilitate other processes and functions. Each of the above identifiedinstructions and applications may correspond to a set of instructionsfor performing one or more functions described above. These instructionsneed not be implemented as separate software programs, procedures, ormodules. Memory device 226 may include additional instructions or fewerinstructions. Furthermore, various functions of server 135 may beimplemented in hardware and/or in software, including in one or moresignal processing and/or application specific integrated circuits. Forexample, server 135 may execute an image processing algorithm toidentify in received images one or more products and/or obstacles, suchas shopping carts, people, and more.

In one embodiment, memory device 226 may store database 140. Database140 may include product type model data 240 (e.g., an imagerepresentation, a list of features, a model obtained by training machinelearning algorithm using training examples, an artificial neuralnetwork, and more) that may be used to identify products in receivedimages; contract-related data 242 (e.g., planograms, promotions data,etc.) that may be used to determine if the placement of products on thestore shelves and/or the promotion execution are consistent withobligations of retail store 105; catalog data 244 (e.g., retail storechain's catalog, retail store's master file, etc.) that may be used tocheck if all product types that should be offered in retail store 105are in fact in the store, if the correct price is displayed next to anidentified product, etc.; inventory data 246 that may be used todetermine if additional products should be ordered from suppliers 115;employee data 248 (e.g., attendance data, records of training provided,evaluation and other performance-related communications, productivityinformation, etc.) that may be used to assign specific employees tocertain tasks; and calendar data 250 (e.g., holidays, national days,international events, etc.) that may be used to determine if a possiblechange in a product model is associated with a certain event. In otherembodiments of the disclosure, database 140 may store additional typesof data or fewer types of data. Furthermore, various types of data maybe stored in one or more memory devices other than memory device 226.

The components and arrangements shown in FIG. 2 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of server 135. For example, not all components may beessential for the operation of server 135 in all cases. Any componentmay be located in any appropriate part of server 135, and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. For example, some serversmay not include some of the elements shown in I/O system 215.

FIG. 3 is a block diagram representation of an example configuration ofcapturing device 125. In one embodiment, capturing device 125 mayinclude a processing device 302, a memory interface 304, a networkinterface 306, and a peripherals interface 308 connected to image sensor310. These components can be separated or can be integrated in one ormore integrated circuits. The various components in capturing device 125can be coupled by one or more communication buses or signal lines (e.g.,bus 300). Different aspects of the functionalities of the variouscomponents in capturing device 125 may be understood from thedescription above regarding components of server 135 having similarfunctionality.

According to one embodiment, network interface 306 may be used tofacilitate communication with server 135. Network interface 306 may bean Ethernet port connected to radio frequency receivers and transmittersand/or optical receivers and transmitters. The specific design andimplementation of network interface 306 depends on the communicationsnetwork(s) over which capturing device 125 is intended to operate. Forexample, in some embodiments, capturing device 125 may include a networkinterface 306 designed to operate over a GSM network, a GPRS network, anEDGE network, a Wi-Fi or WiMax network, a Bluetooth® network, etc. Inanother example, the implementation of network interface 306 may besimilar or identical to the implementation described above for networkinterface 206.

In the example illustrated in FIG. 3, peripherals interface 308 ofcapturing device 125 may be connected to at least one image sensor 310associated with at least one lens 312 for capturing image data in anassociated field of view. In some configurations, capturing device 125may include a plurality of image sensors associated with a plurality oflenses 312. In other configurations, image sensor 310 may be part of acamera included in capturing device 125. According to some embodiments,peripherals interface 308 may also be connected to other sensors (notshown), such as a motion sensor, a light sensor, infrared sensor, soundsensor, a proximity sensor, a temperature sensor, a biometric sensor, orother sensing devices to facilitate related functionalities. Inaddition, a positioning sensor may also be integrated with, or connectedto, capturing device 125. For example, such positioning sensor may beimplemented using one of the following technologies: Global PositioningSystem (GPS), GLObal NAvigation Satellite System (GLONASS), Galileoglobal navigation system, BeiDou navigation system, other GlobalNavigation Satellite Systems (GNSS), Indian Regional NavigationSatellite System (IRNSS), Local Positioning Systems (LPS), Real-TimeLocation Systems (RTLS), Indoor Positioning System (IPS), Wi-Fi basedpositioning systems, cellular triangulation, and so forth. For example,the positioning sensor may be built into mobile capturing device 125,such as smartphone devices. In another example, position software mayallow mobile capturing devices to use internal or external positioningsensors (e.g., connecting via a serial port or Bluetooth).

Consistent with the present disclosure, capturing device 125 may includedigital components that collect data from image sensor 310, transform itinto an image, and store the image on a memory device 314 and/ortransmit the image using network interface 306. In one embodiment,capturing device 125 may be fixedly mountable to a store shelf or toother objects in the retail store (such as walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth). In oneembodiment, capturing device 125 may be split into at least two housingssuch that only image sensor 310 and lens 312 may be visible on the storeshelf, and the rest of the digital components may be located in aseparate housing. An example of this type of capturing device isdescribed below with reference to FIGS. 5-7.

Consistent with the present disclosure, capturing device 125 may usememory interface 304 to access memory device 314. Memory device 314 mayinclude high-speed, random access memory and/or non-volatile memory suchas one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR) to store capturedimage data. Memory device 314 may store operating system instructions316, such as DARWIN, RTXC, LINUX, iOS, UNIX, LINUX, OS X, WINDOWS, or anembedded operating system such as VXWorkS. Operating system 316 caninclude instructions for handling basic system services and forperforming hardware dependent tasks. In some implementations, operatingsystem 316 may include a kernel (e.g., UNIX kernel, LINUX kernel, etc.).In addition, memory device 314 may store capturing instructions 318 tofacilitate processes and functions related to image sensor 310;graphical user interface instructions 320 that enables a user associatedwith capturing device 125 to control the capturing device and/or toacquire images of an area-of-interest in a retail establishment; andapplication instructions 322 to facilitate a process for monitoringcompliance of product placement or other processes.

The components and arrangements shown in FIG. 3 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of capturing device 125. For example, not all componentsare essential for the operation of capturing device 125 in all cases.Any component may be located in any appropriate part of capturing device125, and the components may be rearranged into a variety ofconfigurations while providing the functionality of the disclosedembodiments. For example, some capturing devices may not have lenses,and other capturing devices may include an external memory deviceinstead of memory device 314.

FIGS. 4A-4C illustrate example configurations for capturing image datain retail store 105 according to disclosed embodiments. FIG. 4Aillustrates how an aisle 400 of retail store 105 may be imaged using aplurality of capturing devices 125 fixedly connected to store shelves.FIG. 4B illustrates how aisle 400 of retail store 105 may be imagedusing a handheld communication device. FIG. 4C illustrates how aisle 400of retail store 105 may be imaged by robotic devices equipped withcameras.

With reference to FIG. 4A and consistent with the present disclosure,retail store 105 may include a plurality of capturing devices 125fixedly mounted (for example, to store shelves, walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth) andconfigured to collect image data. As depicted, one side of an aisle 400may include a plurality of capturing devices 125 (e.g., 125A, 125B, and125C) fixedly mounted thereon and directed such that they may captureimages of an opposing side of aisle 400. The plurality of capturingdevices 125 may be connected to an associated mobile power source (e.g.,one or more batteries), to an external power supply (e.g., a powergrid), obtain electrical power from a wireless power transmissionsystem, and so forth. As depicted in FIG. 4A, the plurality of capturingdevices 125 may be placed at different heights and at least theirvertical fields of view may be adjustable. Generally, both sides ofaisle 400 may include capturing devices 125 in order to cover both sidesof aisle 400.

Differing numbers of capturing devices 125 may be used to cover shelvingunit 402. In addition, there may be an overlap region in the horizontalfield of views of some of capturing devices 125. For example, thehorizontal fields of view of capturing devices (e.g., adjacent capturingdevices) may at least partially overlap with one another. In anotherexample, one capturing device may have a lower field of view than thefield of view of a second capturing device, and the two capturingdevices may have at least partially overlapping fields of view.According to one embodiment, each capturing device 125 may be equippedwith network interface 306 for communicating with server 135. In oneembodiment, the plurality of capturing devices 125 in retail store 105may be connected to server 135 via a single WLAN. Network interface 306may transmit information associated with a plurality of images capturedby the plurality of capturing devices 125 for analysis purposes. In oneexample, server 135 may determine an existence of an occlusion event(such as, by a person, by store equipment, such as a ladder, cart, etc.)and may provide a notification to resolve the occlusion event. Inanother example, server 135 may determine if a disparity exists betweenat least one contractual obligation and product placement as determinedbased on automatic analysis of the plurality of images. The transmittedinformation may include raw images, cropped images, processed imagedata, data about products identified in the images, and so forth.Network interface 306 may also transmit information identifying thelocation of the plurality capturing devices 125 in retail store 105.

With reference to FIG. 4B and consistent with the present disclosure,server 135 may receive image data captured by users 120. In a firstembodiment, server 135 may receive image data acquired by storeemployees. In one implementation, a handheld device of a store employee(e.g., capturing device 125D) may display a real-time video streamcaptured by the image sensor of the handheld device. The real-time videostream may be augmented with markings identifying to the store employeean area-of-interest that needs manual capturing of images. One of thesituations in which manual image capture may be desirable may occurwhere the area-of-interest is outside the fields of view of a pluralityof cameras fixedly connected to store shelves in aisle 400. In othersituations, manual capturing of images of an area-of-interest may bedesirable when a current set of acquired images is out of date (e.g.,obsolete in at least one respect) or of poor quality (e.g., lackingfocus, obstacles, lesser resolution, lack of light, etc.). Additionaldetails of this embodiment are described in Applicant's InternationalPatent Application No. PCT/IB2018/001107, which is incorporated hereinby reference.

In a second embodiment, server 135 may receive image data acquired bycrowd sourcing. In one exemplary implementation, server 135 may providea request to a detected mobile device for an updated image of thearea-of-interest in aisle 400. The request may include an incentive(e.g., $2 discount) to user 120 for acquiring the image. In response tothe request, user 120 may acquire and transmit an up-to-date image ofthe area-of-interest. After receiving the image from user 120, server135 may transmit the accepted incentive or agreed upon reward to user120. The incentive may comprise a text notification and a redeemablecoupon. In some embodiments, the incentive may include a redeemablecoupon for a product associated with the area-of-interest. Server 135may generate image-related data based on aggregation of data from imagesreceived from crowd sourcing and from images received from a pluralityof cameras fixedly connected to store shelves. Additional details ofthis embodiment are described in Applicant's International PatentApplication No. PCT/IB2017/000919, which is incorporated herein byreference.

With reference to FIG. 4C and consistent with the present disclosure,server 135 may receive image data captured by robotic devices withcameras traversing in aisle 400. The present disclosure is not limitedto the type of robotic devices used to capture images of retail store105. In some embodiments, the robotic devices may include a robot on atrack (e.g., a Cartesian robot configured to move along an edge of ashelf or in parallel to a shelf, such as capturing device 125E), a drone(e.g., capturing device 125F), and/or a robot that may move on the floorof the retail store (e.g., a wheeled robot such as capturing device125G, a legged robot, a snake-like robot, etc.). The robotic devices maybe controlled by server 135 and may be operated remotely orautonomously. In one example, server 135 may instruct capturing device125E to perform periodic scans at times when no customers or otherobstructions are identified in aisle 400. Specifically, capturing device125E may be configured to move along store shelf 404 and to captureimages of products placed on store shelf 404, products placed on storeshelf 406, or products located on shelves opposite store shelf (e.g.,store shelf 408). In another example, server 135 may instruct capturingdevice 125F to perform a scan of all the area of retail store 105 beforethe opening hour. In another example, server 135 may instruct capturingdevice 125G to capture a specific area-of-interest, similar as describedabove with reference to receiving images acquired by the storeemployees. In some embodiments, robotic capturing devices (such as 125Fand 125G) may include an internal processing unit that may allow them tonavigate autonomously within retail store 105. For example, the roboticcapturing devices may use input from sensors (e.g., image sensors, depthsensors, proximity sensors, etc.), to avoid collision with objects orpeople, and to complete the scan of the desired area of retail store105.

As discussed above with reference to FIG. 4A, the image datarepresentative of products displayed on store shelves may be acquired bya plurality of stationary capturing devices 125 fixedly mounted in theretail store. One advantage of having stationary image capturing devicesspread throughout retail store 105 is the potential for acquiringproduct images from set locations and on an ongoing basis such thatup-to-date product status may be determined for products throughout aretail store at any desired periodicity (e.g., in contrast to a movingcamera system that may acquire product images more infrequently).However, there may be certain challenges in this approach. The distancesand angles of the image capturing devices relative to the capturedproducts should be selected such as to enable adequate productidentification, especially when considered in view of image sensorresolution and/or optics specifications. For example, a capturing deviceplaced on the ceiling of retail store 105 may have sufficientresolutions and optics to enable identification of large products (e.g.,a pack of toilet paper), but may be insufficient for identifying smallerproducts (e.g., deodorant packages). The image capturing devices shouldnot occupy shelf space that is reserved for products for sale. The imagecapturing devices should not be positioned in places where there is alikelihood that their fields of view will be regularly blocked bydifferent objects. The image capturing devices should be able tofunction for long periods of time with minimum maintenance. For example,a requirement for frequent replacement of batteries may render certainimage acquisition systems cumbersome to use, especially where many imageacquisition devices are in use throughout multiple locations in a retailstore and across multiple retail stores. The image capturing devicesshould also include processing capabilities and transmissioncapabilities for providing real time or near real time image data aboutproducts. The disclosed image acquisition systems address thesechallenges.

FIG. 5A illustrates an example of a system 500 for acquiring images ofproducts in retail store 105. Throughout the disclosure, capturingdevice 125 may refer to a system, such as system 500 shown in FIG. 5A.As shown, system 500 may include a first housing 502 configured forlocation on a retail shelving unit (e.g., as illustrated in FIG. 5B),and a second housing 504 configured for location on the retail shelvingunit separate from first housing 502. The first and the second housingmay be configured for mounting on the retail shelving unit in anysuitable way (e.g., screws, bolts, clamps, adhesives, magnets,mechanical means, chemical means, etc.). In some embodiments, firsthousing 502 may include an image capture device 506 (e.g., a cameramodule that may include image sensor 310) and second housing 504 mayinclude at least one processor (e.g., processing device 302) configuredto control image capture device 506 and also to control a networkinterface (e.g., network interface 306) for communicating with a remoteserver (e.g., server 135).

System 500 may also include a data conduit 508 extending between firsthousing 502 and second housing 504. Data conduit 508 may be configuredto enable transfer of control signals from the at least one processor toimage capture device 506 and to enable collection of image data acquiredby image capture device 506 for transmission by the network interface.Consistent with the present disclosure, the term “data conduit” mayrefer to a communications channel that may include either a physicaltransmission medium such as a wire or a logical connection over amultiplexed medium such as a radio channel In some embodiments, dataconduit 508 may be used for conveying image data from image capturedevice 506 to at least one processor located in second housing 504.Consistent with one implementation of system 500, data conduit 508 mayinclude flexible printed circuits and may have a length of at leastabout 5 cm, at least about 10 cm, at least about 15 cm, etc. The lengthof data conduit 508 may be adjustable to enable placement of firsthousing 502 separately from second housing 504. For example, in someembodiments, data conduit may be retractable within second housing 504such that the length of data conduit exposed between first housing 502and second housing 504 may be selectively adjusted.

In one embodiment, the length of data conduit 508 may enable firsthousing 502 to be mounted on a first side of a horizontal store shelffacing the aisle (e.g., store shelf 510 illustrated in FIG. 5B) andsecond housing 504 to be mounted on a second side of store shelf 510that faces the direction of the ground (e.g., an underside of a storeshelf). In this embodiment, data conduit 508 may be configured to bendaround an edge of store shelf 510 or otherwise adhere/follow contours ofthe shelving unit. For example, a first portion of data conduit 508 maybe configured for location on the first side of store shelf 510 (e.g., aside facing an opposing retail shelving unit across an aisle) and asecond portion of data conduit 508 may be configured for location on asecond side of store shelf 510 (e.g., an underside of the shelf, whichin some cases may be orthogonal to the first side). The second portionof data conduit 508 may be longer than the first portion of data conduit508. Consistent with another embodiment, data conduit 508 may beconfigured for location within an envelope of a store shelf. Forexample, the envelope may include the outer boundaries of a channellocated within a store shelf, a region on an underside of an L-shapedstore shelf, a region between two store shelves, etc. Consistent withanother implementation of system 500 discussed below, data conduit 508may include a virtual conduit associated with a wireless communicationslink between first housing 502 and second housing 504.

FIG. 5B illustrates an exemplary configuration for mounting firsthousing 502 on store shelf 510. Consistent with the present disclosure,first housing 502 may be placed on store shelf 510, next to or embeddedin a plastic cover that may be used for displaying prices.Alternatively, first housing 502 may be placed or mounted on any otherlocation in retail store 105. For example, first housing 502 may beplaced or mounted on the walls, on the ceiling, on refrigerator units,on display units, and more. The location and/or orientation of firsthousing 502 may be selected such that a field of view of image capturedevice 506 may cover at least a portion of an opposing retail shelvingunit. Consistent with the present disclosure, image capture device 506may have a view angle of between 50 and 80 degrees, about 62 degrees,about 67 degrees, or about 75 degrees. Consistent with the presentdisclosure, image capture device 506 may include an image sensor havingsufficient image resolution to enable detection of text associated withlabels on an opposing retail shelving unit. In one embodiment, the imagesensor may include m*n pixels. For example, image capture device 506 mayhave an 8 MP image sensor that includes an array of 3280*2464 pixels.Each pixel may include at least one photo-voltaic cell that converts thephotons of the incident light to an electric signal. The electricalsignal may be converted to digital data by an A/D converter andprocessed by the image processor (ISP). In one embodiment, the imagesensor of image capture device 506 may be associated with a pixel sizeof between 1.1×1.1 um² and 1.7×1.7 um², for example, 1.4×1.4 um².

Consistent with the present disclosure, image capture device 506 may beassociated with a lens (e.g., lens 312) having a fixed focal lengthselected according to a distance expected to be encountered betweenretail shelving units on opposite sides of an aisle (e.g., distance d1shown in FIG. 6A) and/or according to a distance expected to beencountered between a side of a shelving unit facing the aisle on oneside of an aisle and a side of a shelving unit facing away of the aisleon the other side of the aisle (e.g., distance d2 shown in FIG. 6A). Thefocal length may also be based on any other expected distance betweenthe image acquisition device and products to be imaged. As used herein,the term “focal length” refers to the distance from the optical centerof the lens to a point where objects located at the point aresubstantially brought into focus. In contrast to zoom lenses, in fixedlenses the focus is not adjustable. The focus is typically set at thetime of lens design and remains fixed. In one embodiment, the focallength of lens 312 may be selected based on the distance between twosides of aisles in the retail store (e.g., distance d1, distance d2, andso forth). In some embodiments, image capture device 506 may include alens with a fixed focal length having a fixed value between 2.5 mm and4.5 mm, such as about 3.1 mm, about 3.4 mm, about 3.7 mm. For example,when distance d1 between two opposing retail shelving units is about 2meters, the focal length of the lens may be about 3.6 mm. Unlessindicated otherwise, the term “about” with regards to a numeric value isdefined as a variance of up to 5% with respect to the stated value. Ofcourse, image capture devices having non-fixed focal lengths may also beused depending on the requirements of certain imaging environments, thepower and space resources available, etc.

FIG. 5C illustrates an exploded view of second housing 504. In someembodiments, the network interface located in second housing 504 (e.g.,network interface 306) may be configured to transmit to server 135information associated with a plurality of images captured by imagecapture device 506. For example, the transmitted information may be usedto determine if a disparity exists between at least one contractualobligation (e.g. planogram) and product placement. In one example, thenetwork interface may support transmission speeds of 0.5 Mb/s, 1 Mb/s, 5Mb/s, or more. Consistent with the present disclosure, the networkinterface may allow different modes of operations to be selected, suchas: high-speed, slope-control, or standby. In high-speed mode,associated output drivers may have fast output rise and fall times tosupport high-speed bus rates; in slope-control, the electromagneticinterference may be reduced and the slope (i.e., the change of voltageper unit of time) may be proportional to the current output; and instandby mode, the transmitter may be switched off and the receiver mayoperate at a lower current.

Consistent with the present disclosure, second housing 504 may include apower port 512 for conveying energy from a power source to first housing502. In one embodiment, second housing 504 may include a section for atleast one mobile power source 514 (e.g., in the depicted configurationthe section is configured to house four batteries). The at least onemobile power source may provide sufficient power to enable image capturedevice 506 to acquire more than 1,000 pictures, more than 5,000pictures, more than 10,000 pictures, or more than 15,000 pictures, andto transmit them to server 135. In one embodiment, mobile power source514 located in a single second housing 504 may power two or more imagecapture devices 506 mounted on the store shelf. For example, as depictedin FIGS. 6A and 6B, a single second housing 504 may be connected to aplurality of first housings 502 with a plurality of image capturedevices 506 covering different (overlapping or non-overlapping) fieldsof view. Accordingly, the two or more image capture devices 506 may bepowered by a single mobile power source 514 and/or the data captured bytwo or more image capture devices 506 may be processed to generate apanoramic image by a single processing device located in second housing504. In addition to mobile power source 514 or as an alternative tomobile power source 514, second housing 504 may also be connected to anexternal power source. For example, second housing 504 may be mounted toa store shelf and connected to an electric power grid. In this example,power port 512 may be connected to the store shelf through a wire forproviding electrical power to image capture device 506. In anotherexample, a retail shelving unit or retail store 105 may include awireless power transmission system, and power port 512 may be connectedto a device configured to obtain electrical power from the wirelesspower transmission system. In addition, as discussed below, system 500may use power management policies to reduce the power consumption. Forexample, system 500 may use selective image capturing and/or selectivetransmission of images to reduce the power consumption or conservepower.

FIG. 6A illustrates a schematic diagram of a top view of aisle 600 inretail store 105 with multiple image acquisition systems 500 (e.g.,500A, 500B, 500C, 500D, and 500E) deployed thereon for acquiring imagesof products. Aisle 600 may include a first retail shelving unit 602 anda second retail shelving unit 604 that opposes first retail shelvingunit 602. In some embodiments, different numbers of systems 500 may bemounted on opposing retail shelving units. For example, system 500A(including first housing 502A, second housing 504A, and data conduit508A), system 500B (including first housing 502B second housing 504B,and data conduit 508B), and system 500C (including first housing 502C,second housing 504C, and data conduit 508C) may be mounted on firstretail shelving unit 602; and system 500D (including first housing502D1, first housing 502D2, second housing 504D, and data conduits 508D1and 508D2) and system 500E (including first housing 502E1, first housing502E2, second housing 504E, and data conduits 508E1 and 508E2) may bemounted on second retail shelving unit 604. Consistent with the presentdisclosure, image capture device 506 may be configured relative to firsthousing 502 such that an optical axis of image capture device 506 isdirected toward an opposing retail shelving unit when first housing 502is fixedly mounted on a retail shelving unit. For example, optical axis606 of the image capture device associated with first housing 502B maybe directed towards second retail shelving unit 604 when first housing502B is fixedly mounted on first retail shelving unit 602. A singleretail shelving unit may hold a number of systems 500 that include aplurality of image capturing devices. Each of the image capturingdevices may be associated with a different field of view directed towardthe opposing retail shelving unit. Different vantage points ofdifferently located image capture devices may enable image acquisitionrelative to different sections of a retail shelf. For example, at leastsome of the plurality of image capturing devices may be fixedly mountedon shelves at different heights. Examples of such a deployment areillustrated in FIGS. 4A and 6B.

As shown in FIG. 6A each first housing 502 may be associated with a dataconduit 508 that enables exchanging of information (e.g., image data,control signals, etc.) between the at least one processor located insecond housing 504 and image capture device 506 located in first housing502. In some embodiments, data conduit 508 may include a wiredconnection that supports data-transfer and may be used to power imagecapture device 506 (e.g., data conduit 508A, data conduit 508B, dataconduit 508D1, data conduit 508D2, data conduit 508E1, and data conduit508E2). Consistent with these embodiments, data conduit 508 may complywith a wired standard such as USB, Micro-USB, HDMI, Micro-HDMI,Firewire, Apple, etc. In other embodiments, data conduit 508 may be awireless connection, such as a dedicated communications channel betweenthe at least one processor located in second housing 504 and imagecapture device 506 located in first housing 502 (e.g., data conduit508C). In one example, the communications channel may be established bytwo Near Field Communication (NFC) transceivers. In other examples,first housing 502 and second housing 504 may include interface circuitsthat comply with other short-range wireless standards such as Bluetooth,WiFi, ZigBee, etc.

In some embodiments of the disclosure, the at least one processor ofsystem 500 may cause at least one image capture device 506 toperiodically capture images of products located on an opposing retailshelving unit (e.g., images of products located on a shelf across anaisle from the shelf on which first housing 502 is mounted). The term“periodically capturing images” includes capturing an image or images atpredetermined time intervals (e.g., every minute, every 30 minutes,every 150 minutes, every 300 minutes, etc.), capturing video, capturingan image every time a status request is received, and/or capturing animage subsequent to receiving input from an additional sensor, forexample, an associated proximity sensor. Images may also be capturedbased on various other triggers or in response to various other detectedevents. In some embodiments, system 500 may receive an output signalfrom at least one sensor located on an opposing retail shelving unit.For example, system 500B may receive output signals from a sensingsystem located on second retail shelving unit 604. The output signalsmay be indicative of a sensed lifting of a product from second retailshelving unit 604 or a sensed positioning of a product on second retailshelving unit 604. In response to receiving the output signal from theat least one sensor located on second retail shelving unit 604, system500B may cause image capture device 506 to capture one or more images ofsecond retail shelving unit 604. Additional details on a sensing system,including the at least one sensor that generates output signalsindicative of a sensed lifting of a product from an opposing retailshelving unit, is discussed below with reference to FIGS. 8-10.

Consistent with embodiments of the disclosure, system 500 may detect anobject 608 in a selected area between first retail shelving unit 602 andsecond retail shelving unit 604. Such detection may be based on theoutput of one or more dedicated sensors (e.g., motion detectors, etc.)and/or may be based on image analysis of one or more images acquired byan image acquisition device. Such images, for example, may include arepresentation of a person or other object recognizable through variousimage analysis techniques (e.g., trained neural networks, Fouriertransform analysis, edge detection, filters, face recognition, etc.).The selected area may be associated with distance d1 between firstretail shelving unit 602 and second retail shelving unit 604. Theselected area may be within the field of view of image capture device506 or an area where the object causes an occlusion of a region ofinterest (such as a shelf, a portion of a shelf being monitored, andmore). Upon detecting object 608, system 500 may cause image capturedevice 506 to forgo image acquisition while object 608 is within theselected area. In one example, object 608 may be an individual, such asa customer or a store employee. In another example, detected object 608may be an inanimate object, such as a cart, box, carton, one or moreproducts, cleaning robots, etc. In the example illustrated in FIG. 6A,system 500A may detect that object 608 has entered into its associatedfield of view (e.g., using a proximity sensor) and may instruct imagecapturing device 506 to forgo image acquisition. In alternativeembodiments, system 500 may analyze a plurality of images acquired byimage capture device 506 and identify at least one image of theplurality of images that includes a representation of object 608.Thereafter, system 500 may avoid transmission of at least part of the atleast one identified image and/or information based on the at least oneidentified image to server 135.

As shown in FIG. 6A, the at least one processor contained in a secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., systems 500D and500E). Controlling image capturing device 506 may include instructingimage capturing device 506 to capture an image and/or transmit capturedimages to a remote server (e.g., server 135). In some cases, each of theplurality of image capture devices 506 may have a field of view that atleast partially overlaps with a field of view of at least one otherimage capture device 506 from among plurality of image capture devices506. In one embodiment, the plurality of image capture devices 506 maybe configured for location on one or more horizontal shelves and may bedirected to substantially different areas of the opposing first retailshelving unit. In this embodiment, the at least one processor maycontrol the plurality of image capture devices such that each of theplurality of image capture devices may capture an image at a differenttime. For example, system 500E may have a second housing 504E with atleast one processor that may instruct a first image capturing devicecontained in first housing 502E1 to capture an image at a first time andmay instruct a second image capturing device contained in first housing502E2 to capture an image at a second time which differs from the firsttime. Capturing images in different times (or forwarding them to the atleast one processor at different times) may assist in processing theimages and writing the images in the memory associated with the at leastone processor.

FIG. 6B illustrates a perspective view assembly diagram depicting aportion of a retail shelving unit 620 with multiple systems 500 (e.g.,500F, 500G, 500H, 500I, and 500J) deployed thereon for acquiring imagesof products. Retail shelving unit 620 may include horizontal shelves atdifferent heights. For example, horizontal shelves 622A, 622B, and 622Care located below horizontal shelves 622D, 622E, and 622F. In someembodiments, a different number of systems 500 may be mounted on shelvesat different heights. For example, system 500F (including first housing502F and second housing 504F), system 500G (including first housing 502Gand second housing 504G), and system 500H (including first housing 502Hand second housing 504H) may be mounted on horizontal shelves associatedwith a first height; and system 500I (including first housing 502I,second housing 504I, and a projector 632) and system 500J (includingfirst housing 502J1, first housing 502J2, and second housing 504J) maybe mounted on horizontal shelves associated with a second height. Insome embodiments, retail shelving unit 620 may include a horizontalshelf with at least one designated place (not shown) for mounting ahousing of image capturing device 506. The at least one designated placemay be associated with connectors such that first housing 502 may befixedly mounted on a side of horizontal shelf 622 facing an opposingretail shelving unit using the connectors.

Consistent with the present disclosure, system 500 may be mounted on aretail shelving unit that includes at least two adjacent horizontalshelves (e.g., shelves 622A and 622B) forming a substantially continuoussurface for product placement. The store shelves may include standardstore shelves or customized store shelves. A length of each store shelf622 may be at least 50 cm, less than 200 cm, or between 75 cm to 175 cm.In one embodiment, first housing 502 may be fixedly mounted on theretail shelving unit in a slit between two adjacent horizontal shelves.For example, first housing 502G may be fixedly mounted on retailshelving unit 620 in a slit between horizontal shelf 622B and horizontalshelf 622C. In another embodiment, first housing 502 may be fixedlymounted on a first shelf and second housing 504 may be fixedly mountedon a second shelf. For example, first housing 502I may be mounted onhorizontal shelf 622D and second housing 504I may be mounted onhorizontal shelf 622E. In another embodiment, first housing 502 may befixedly mounted on a retail shelving unit on a first side of ahorizontal shelf facing the opposing retail shelving unit and secondhousing 504 may be fixedly mounted on retail shelving unit 620 on asecond side of the horizontal shelf orthogonal to the first side. Forexample, first housing 502H may mounted on a first side 624 ofhorizontal shelf 622C next to a label and second housing 504H may bemounted on a second side 626 of horizontal shelf 622C that faces down(e.g., towards the ground or towards a lower shelf). In anotherembodiment, second housing 504 may be mounted closer to the back of thehorizontal shelf than to the front of the horizontal shelf. For example,second housing 504H may be fixedly mounted on horizontal shelf 622C onsecond side 626 closer to third side 628 of the horizontal shelf 622Cthan to first side 624. Third side 628 may be parallel to first side624. As mentioned above, data conduit 508 (e.g., data conduit 508H) mayhave an adjustable or selectable length for extending between firsthousing 502 and second housing 504. In one embodiment, when firsthousing 502H is fixedly mounted on first side 624, the length of dataconduit 508H may enable second housing 604H to be fixedly mounted onsecond side 626 closer to third side 628 than to first side 624.

As mentioned above, at least one processor contained in a single secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., system 500J). Insome embodiments, the plurality of image capture devices 506 may beconfigured for location on a single horizontal shelf and may be directedto substantially the same area of the opposing first retail shelvingunit (e.g., system 500D in FIG. 6A). In these embodiments, the imagedata acquired by the first image capture device and the second imagecapture device may enable a calculation of depth information (e.g.,based on image parallax information) associated with at least oneproduct positioned on an opposing retail shelving unit. For example,system 500J may have single second housing 504J with at least oneprocessor that may control a first image capturing device contained infirst housing 502J1 and a second image capturing device contained infirst housing 502J2. The distance d3 between the first image capturedevice contained in first housing 502J1 and the second image capturedevice contained in first housing 502J2 may be selected based on thedistance between retail shelving unit 620 and the opposing retailshelving unit (e.g., similar to d1 and/or d2). For example, distance d3may be at least 5 cm, at least 10 cm, at least 15 cm, less than 40 cm,less than 30 cm, between about 5 cm to about 20 cm, or between about 10cm to about 15 cm. In another example, d3 may be a function of d1 and/ord2, a linear function of d1 and/or d2, a function of d1*log(d1) and/ord2*log(d2) such as a1* d1*log(d1) for some constant a1, and so forth.The data from the first image capturing device contained in firsthousing 502J1 and the second image capturing device contained in firsthousing 502J2 may be used to estimate the number of products on a storeshelf of retail shelving unit 602. In related embodiments, system 500may control a projector (e.g., projector 632) and image capture device506 that are configured for location on a single store shelf or on twoseparate store shelves. For example, projector 632 may be mounted onhorizontal shelf 622E and image capture device 506I may be mounted onhorizontal shelf 622D. The image data acquired by image capture device506 (e.g., included in first housing 502I) may include reflections oflight patterns projected from projector 632 on the at least one productand/or the opposing retail shelving unit and may enable a calculation ofdepth information associated with at least one product positioned on theopposing retail shelving unit. The distance between projector 632 andthe image capture device contained in first housing 502I may be selectedbased on the distance between retail shelving unit 620 and the opposingretail shelving unit (e.g., similar to d1 and/or d2). For example, thedistance between the projector and the image capture device may be atleast 5 cm, at least 10 cm, at least 15 cm, less than 40 cm, less than30 cm, between about 5 cm to about 20 cm, or between about 10 cm toabout 15 cm. In another example, the distance between the projector andthe image capture device may be a function of d1 and/or d2, a linearfunction of d1 and/or d2, a function of d1*log(d1) and/or d2*log(d2)such as a1* d1*log(d1) for some constant a1, and so forth.

Consistent with the present disclosure, a central communication device630 may be located in retail store 105 and may be configured tocommunicate with server 135 (e.g., via an Internet connection). Thecentral communication device may also communicate with a plurality ofsystems 500 (for example, less than ten, ten, eleven, twelve, more thantwelve, and so forth). In some cases, at least one system of theplurality of systems 500 may be located in proximity to centralcommunication device 630. In the illustrated example, system 500F may belocated in proximity to central communication device 630. In someembodiments, at least some of systems 500 may communicate directly withat least one other system 500. The communications between some of theplurality of systems 500 may happen via a wired connection, such as thecommunications between system 500J and system 500I and thecommunications between system 500H and system 500G. Additionally oralternatively, the communications between some of the plurality ofsystems 500 may occur via a wireless connection, such as thecommunications between system 500G and system 500F and thecommunications between system 500I and system 500F. In some examples, atleast one system 500 may be configured to transmit captured image data(or information derived from the captured image data) to centralcommunication device 630 via at least two mediating systems 500, atleast three mediating systems 500, at least four mediating systems 500,or more. For example, system 500J may convey captured image data tocentral communication device 630 via system 500I and system 500F.

Consistent with the present disclosure, two (or more) systems 500 mayshare information to improve image acquisition. For example, system 500Jmay be configured to receive from a neighboring system 500I informationassociated with an event that system 500I had identified, and controlimage capture device 506 based on the received information. For example,system 500J may forgo image acquisition based on an indication fromsystem 500I that an object has entered or is about to enter its field ofview. Systems 500I and 500J may have overlapping fields of view ornon-overlapping fields of view. In addition, system 500J may alsoreceive (from system 500I) information that originates from centralcommunication device 630 and control image capture device 506 based onthe received information. For example, system 500I may receiveinstructions from central communication device 630 to capture an imagewhen suppler 115 inquiries about a specific product that is placed in aretail unit opposing system 500I. In some embodiments, a plurality ofsystems 500 may communicate with central communication device 630. Inorder to reduce or avoid network congestion, each system 500 mayidentify an available transmission time slot. Thereafter, each system500 may determine a default time slot for future transmissions based onthe identified transmission time slot.

FIG. 6C provides a diagrammatic representation of a retail shelving unit640 being captured by multiple systems 500 (e.g., system 500K and system500L) deployed on an opposing retail shelving unit (not shown). FIG. 6Cillustrates embodiments associated with the process of installingsystems 500 in retail store 105. To facilitate the installation ofsystem 500, each first housing 502 (e.g., first housing 502K) mayinclude an adjustment mechanism 642 for setting a field of view 644 ofimage capture device 506K such that the field of view 644 will at leastpartially encompass products placed both on a bottom shelf of retailshelving unit 640 and on a top shelf of retail shelving unit 640. Forexample, adjustment mechanism 642 may enable setting the position ofimage capture device 506K relative to first housing 502K. Adjustmentmechanism 642 may have at least two degrees of freedom to separatelyadjust manually (or automatically) the vertical field of view and thehorizontal field of view of image capture device 506K. In oneembodiment, the angle of image capture device 506K may be measured usingposition sensors associated with adjustment mechanism 642, and themeasured orientation may be used to determine if image capture device506K is positioned in the right direction. In one example, the output ofthe position sensors may be displayed on a handheld device of anemployee installing image capturing device 506K. Such an arrangement mayprovide the employee/installer with real time visual feedbackrepresentative of the field of view of an image acquisition device beinginstalled.

In addition to adjustment mechanism 642, first housing 502 may include afirst physical adapter (not shown) configured to operate with multipletypes of image capture device 506 and a second physical adapter (notshown) configured to operate with multiple types of lenses. Duringinstallation, the first physical adapter may be used to connect asuitable image capture device 506 to system 500 according to the levelof recognition requested (e.g., detecting a barcode from products,detecting text and price from labels, detecting different categories ofproducts, etc.). Similarly, during installation, the second physicaladapter may be used to associate a suitable lens to image capture device506 according to the physical conditions at the store (e.g., thedistance between the aisles, the horizontal field of view required fromimage capture device 506, and/or the vertical field of view requiredfrom image capture device 506). The second physical adapter provides theemployee/installer the ability to select the focal length of lens 312during installation according to the distance between retail shelvingunits on opposite sides of an aisle (e.g., distance d1 and/or distanced2 shown in FIG. 6A). In some embodiments, adjustment mechanism 642 mayinclude a locking mechanism to reduce the likelihood of unintentionalchanges in the field of view of image capture device 506. Additionallyor alternatively, the at least one processor contained in second housing504 may detect changes in the field of view of image capture device 506and issue a warning when a change is detected, when a change larger thana selected threshold is detected, when a change is detected for aduration longer than a selected threshold, and so forth.

In addition to adjustment mechanism 642 and the different physicaladapters, system 500 may modify the image data acquired by image capturedevice 506 based on at least one attribute associated with opposingretail shelving unit 640. Consistent with the present disclosure, the atleast one attribute associated with retail shelving unit 640 may includea lighting condition, the dimensions of opposing retail shelving unit640, the size of products displayed on opposing retail shelving unit640, the type of labels used on opposing retail shelving unit 640, andmore. In some embodiments, the attribute may be determined, based onanalysis of one or more acquired images, by at least one processorcontained in second housing 504. Alternatively, the attribute may beautomatically sensed and conveyed to the at least one processorcontained in second housing 504. In one example, the at least oneprocessor may change the brightness of captured images based on thedetected light conditions. In another example, the at least oneprocessor may modify the image data by cropping the image such that itwill include only the products on retail shelving unit (e.g., not toinclude the floor or the ceiling), only area of the shelving unitrelevant to a selected task (such as planogram compliance check), and soforth.

Consistent with the present disclosure, during installation, system 500may enable real-time display 646 of field of view 644 on a handhelddevice 648 of a user 650 installing image capturing device 506K. In oneembodiment, real-time display 646 of field of view 644 may includeaugmented markings 652 indicating a location of a field of view 654 ofan adjacent image capture device 506L. In another embodiment, real-timedisplay 646 of field of view 644 may include augmented markings 656indicating a region of interest in opposing retail shelving unit 640.The region of interest may be determined based on a planogram,identified product type, and/or part of retail shelving unit 640. Forexample, the region of interest may include products with a greaterlikelihood of planogram incompliance. In addition, system 500K mayanalyze acquired images to determine if field of view 644 includes thearea that image capturing device 506K is supposed to monitor (forexample, from labels on opposing retail shelving unit 640, products onopposing retail shelving unit 640, images captured from other imagecapturing devices that may capture other parts of opposing retailshelving unit 640 or capture the same part of opposing retail shelvingunit 640 but in a lower resolution or at a lower frequency, and soforth). In additional embodiments, system 500 may further comprise anindoor location sensor which may help determine if the system 500 ispositioned at the right location in retail store 105.

In some embodiments, an anti-theft device may be located in at least oneof first housing 502 and second housing 504. For example, the anti-theftdevice may include a specific RF label or a pin-tag radio-frequencyidentification device, which may be the same or similar to a type ofanti-theft device that is used by retail store 105 in which system 500is located. The RF label or the pin-tag may be incorporated within thebody of first housing 502 and second housing 504 and may not be visible.In another example, the anti-theft device may include a motion sensorwhose output may be used to trigger an alarm in the case of motion ordisturbance, in case of motion that is above a selected threshold, andso forth.

FIG. 7A includes a flowchart representing an exemplary method 700 foracquiring images of products in retail store 105 in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method can be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 702, the method includes fixedly mounting on first retailshelving unit 602 at least one first housing 502 containing at least oneimage capture device 506 such that an optical axis (e.g., optical axis606) of at least one image capture device 506 is directed to secondretail shelving unit 604. In one embodiment, fixedly mounting firsthousing 502 on first retail shelving unit 602 may include placing firsthousing 502 on a side of store shelf 622 facing second retail shelvingunit 604. In another embodiment, fixedly mounting first housing 502 onretail shelving unit 602 may include placing first housing 502 in a slitbetween two adjacent horizontal shelves. In some embodiments, the methodmay further include fixedly mounting on first retail shelving unit 602at least one projector (such as projector 632) such that light patternsprojected by the at least one projector are directed to second retailshelving unit 604. In one embodiment, the method may include mountingthe at least one projector to first retail shelving unit 602 at aselected distance to first housing 502 with image capture device 506. Inone embodiment, the selected distance may be at least 5 cm, at least 10cm, at least 15 cm, less than 40 cm, less than 30 cm, between about 5 cmto about 20 cm, or between about 10 cm to about 15 cm. In oneembodiment, the selected distance may be calculated according to adistance between to first retail shelving unit 602 and second retailshelving unit 604, such as d1 and/or d2, for example selecting thedistance to be a function of d1 and/or d2, a linear function of d1and/or d2, a function of d1*log(d1) and/or d2*log(d2) such as a1*d1*log(d1) for some constant a1, and so forth.

At step 704, the method includes fixedly mounting on first retailshelving unit 602 second housing 504 at a location spaced apart from theat least one first housing 502, second housing 504 may include at leastone processor (e.g., processing device 302). In one embodiment, fixedlymounting second housing 504 on the retail shelving unit may includeplacing second housing 504 on a different side of store shelf 622 thanthe side first housing 502 is mounted on.

At step 706, the method includes extending at least one data conduit 508between at least one first housing 502 and second housing 504. In oneembodiment, extending at least one data conduit 508 between at least onefirst housing 502 and second housing 504 may include adjusting thelength of data conduit 508 to enable first housing 502 to be mountedseparately from second housing 504. At step 708, the method includescapturing images of second retail shelving unit 604 using at least oneimage capture device 506 contained in at least one first housing 502(e.g., first housing 502A, first housing 502B, or first housing 502C).In one embodiment, the method further includes periodically capturingimages of products located on second retail shelving unit 604. Inanother embodiment the method includes capturing images of second retailshelving unit 604 after receiving a trigger from at least one additionalsensor in communication with system 500 (wireless or wired).

At step 710, the method includes transmitting at least some of thecaptured images from second housing 504 to a remote server (e.g., server135) configured to determine planogram compliance relative to secondretail shelving unit 604. In some embodiments, determining planogramcompliance relative to second retail shelving unit 604 may includedetermining at least one characteristic of planogram compliance based ondetected differences between the at least one planogram and the actualplacement of the plurality of product types on second retail shelvingunit 604. Consistent with the present disclosure, the characteristic ofplanogram compliance may include at least one of: product facing,product placement, planogram compatibility, price correlation, promotionexecution, product homogeneity, restocking rate, and planogramcompliance of adjacent products.

FIG. 7B provides a flowchart representing an exemplary method 720 foracquiring images of products in retail store 105, in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method can be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 722, at least one processor contained in a second housing mayreceive from at least one image capture device contained in at least onefirst housing fixedly mounted on a retail shelving unit a plurality ofimages of an opposing retail shelving unit. For example, at least oneprocessor contained in second housing 504A may receive from at least oneimage capture device 506 contained in first housing 502A (fixedlymounted on first retail shelving unit 602) a plurality of images ofsecond retail shelving unit 604. The plurality of images may be capturedand collected during a period of time (e.g., a minute, an hour, sixhours, a day, a week, or more).

At step 724, the at least one processor contained in the second housingmay analyze the plurality of images acquired by the at least one imagecapture device. In one embodiment, at least one processor contained insecond housing 504A may use any suitable image analysis technique (forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc.) to identify objects in the plurality of images. In oneexample, the at least one processor contained in second housing 504A maydetermine the number of products located in second retail shelving unit604. In another example, the at least one processor contained in secondhousing 504A may detect one or more objects in an area between firstretail shelving unit 602 and second retail shelving unit 604.

At step 726, the at least one processor contained in the second housingmay identify in the plurality of images a first image that includes arepresentation of at least a portion of an object located in an areabetween the retail shelving unit and the opposing retail shelving unit.In step 728, the at least one processor contained in the second housingmay identify in the plurality of images a second image that does notinclude any object located in an area between the retail shelving unitand the opposing retail shelving unit. In one example, the object in thefirst image may be an individual, such as a customer or a storeemployee. In another example, the object in the first image may be aninanimate object, such as carts, boxes, products, etc.

At step 730, the at least one processor contained in the second housingmay instruct a network interface contained in the second housing,fixedly mounted on the retail shelving unit separate from the at leastone first housing, to transmit the second image to a remote server andto avoid transmission of the first image to the remote server. Inaddition, the at least one processor may issue a notification when anobject blocks the field of view of the image capturing device for morethan a predefined period of time (e.g., at least 30 minutes, at least 75minutes, at least 150 minutes).

Embodiments of the present disclosure may automatically assesscompliance of one or more store shelves with a planogram. For example,embodiments of the present disclosure may use signals from one or moresensors to determine placement of one or more products on store shelves.The disclosed embodiments may also use one or more sensors to determineempty spaces on the store shelves. The placements and empty spaces maybe automatically assessed against a digitally encoded planogram. Aplanogram refers to any data structure or specification that defines atleast one product characteristic relative to a display structureassociated with a retail environment (such as store shelf or area of oneor more shelves). Such product characteristics may include, among otherthings, quantities of products with respect to areas of the shelves,product configurations or product shapes with respect to areas of theshelves, product arrangements with respect to areas of the shelves,product density with respect to areas of the shelves, productcombinations with respect to areas of the shelves, etc. Althoughdescribed with reference to store shelves, embodiments of the presentdisclosure may also be applied to end caps or other displays; bins,shelves, or other organizers associated with a refrigerator or freezerunits; or any other display structure associated with a retailenvironment.

The embodiments disclosed herein may use any sensors configured todetect one or more parameters associated with products (or a lackthereof). For example, embodiments may use one or more of pressuresensors, weight sensors, light sensors, resistive sensors, capacitivesensors, inductive sensors, vacuum pressure sensors, high pressuresensors, conductive pressure sensors, infrared sensors, photo-resistorsensors, photo-transistor sensors, photo-diodes sensors, ultrasonicsensors, or the like. Some embodiments may use a plurality of differentkinds of sensors, for example, associated with the same or overlappingareas of the shelves and/or associated with different areas of theshelves. Some embodiments may use a plurality of sensors configured tobe placed adjacent a store shelf, configured for location on the storeshelf, configured to be attached to, or configured to be integrated withthe store shelf. In some cases, at least part of the plurality ofsensors may be configured to be placed next to a surface of a storeshelf configured to hold products. For example, the at least part of theplurality of sensors may be configured to be placed relative to a partof a store shelf such that the at least part of the plurality of sensorsmay be positioned between the part of a store shelf and products placedon the part of the shelf. In another embodiment, the at least part ofthe plurality of sensors may be configured to be placed above and/orwithin and/or under the part of the shelf.

In one example, the plurality of sensors may include light detectorsconfigured to be located such that a product placed on the part of theshelf may block at least some of the ambient light from reaching thelight detectors. The data received from the light detectors may beanalyzed to detect a product or to identify a product based on the shapeof a product placed on the part of the shelf. In one example, the systemmay identify the product placed above the light detectors based on datareceived from the light detectors that may be indicative of at leastpart of the ambient light being blocked from reaching the lightdetectors. Further, the data received from the light detectors may beanalyzed to detect vacant spaces on the store shelf. For example, thesystem may detect vacant spaces on the store shelf based on the receiveddata that may be indicative of no product being placed on a part of theshelf. In another example, the plurality of sensors may include pressuresensors configured to be located such that a product placed on the partof the shelf may apply detectable pressure on the pressure sensors.Further, the data received from the pressure sensors may be analyzed todetect a product or to identify a product based on the shape of aproduct placed on the part of the shelf. In one example, the system mayidentify the product placed above the pressure sensors based on datareceived from the pressure sensors being indicative of pressure beingapplied on the pressure sensors. In addition, the data from the pressuresensors may be analyzed to detect vacant spaces on the store shelf, forexample based on the readings being indicative of no product beingplaced on a part of the shelf, for example, when the pressure readingsare below a selected threshold. Consistent with the present disclosure,inputs from different types of sensors (such as pressure sensors, lightdetectors, etc.) may be combined and analyzed together, for example todetect products placed on a store shelf, to identify shapes of productsplaced on a store shelf, to identify types of products placed on a storeshelf, to identify vacant spaces on a store shelf, and so forth.

With reference to FIG. 8A and consistent with the present disclosure, astore shelf 800 may include a plurality of detection elements, e.g.,detection elements 801A and 801B. In the example of FIG. 8A, detectionelements 801A and 801B may comprise pressure sensors and/or other typeof sensors for measuring one or more parameters (such as resistance,capacitance, or the like) based on physical contact (or lack thereof)with products, e.g., product 803A and product 803B. Additionally oralternatively, detection elements configured to measure one or moreparameters (such as current induction, magnetic induction, visual orother electromagnetic reflectance, visual or other electromagneticemittance, or the like) may be included to detect products based onphysical proximity (or lack thereof) to products. Consistent with thepresent disclosure, the plurality of detection elements may beconfigured for location on shelf 800. The plurality of detectionelements may be configured to detect placement of products when theproducts are placed above at least part of the plurality of detectionelements. Some embodiments of the disclosure, however, may be performedwhen at least some of the detection elements may be located next toshelf 800 (e.g., for magnetometers or the like), across from shelf 800(e.g., for image sensors or other light sensors, light detection andranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, orthe like), above shelf 800 (e.g., for acoustic sensors or the like),below shelf 800 (e.g., for pressure sensors or the like), or any otherappropriate spatial arrangement. Although depicted as standalone unitsin the example of FIG. 8A, the plurality of detection elements may formpart of a fabric (e.g., a smart fabric or the like), and the fabric maybe positioned on a shelf to take measurements. For example, two or moredetection elements may be integrated together into a single structure(e.g., disposed within a common housing, integrated together within afabric or mat, etc.). In some examples, detection elements (such asdetection elements 801A and 801B) may be placed adjacent to (or placedon) store shelves as described above. Some examples of detectionelements may include pressure sensors and/or light detectors configuredto be placed above and/or within and/or under a store shelf as describedabove.

Detection elements associated with shelf 800 may be associated withdifferent areas of shelf 800. For example, detection elements 801A and801B are associated with area 805A while other detection elements areassociated with area 805B. Although depicted as rows, areas 805A and805B may comprise any areas of shelf 800, whether contiguous (e.g., asquare, a rectangular, or other regular or irregular shape) or not(e.g., a plurality of rectangles or other regular and/or irregularshapes). Such areas may also include horizontal regions between shelves(as shown in FIG. 8A) or may include vertical regions that include areaof multiple different shelves (e.g., columnar regions spanning overseveral different horizontally arranged shelves). In some examples, theareas may be part of a single plane. In some examples, each area may bepart of a different plane. In some examples, a single area may be partof a single plane or be divided across multiple planes.

One or more processors (e.g., processing device 202) configured tocommunicate with the detection elements (e.g., detection elements 801Aand 801B) may detect first signals associated with a first area (e.g.,areas 805A and/or 805B) and second signals associated with a secondarea. In some embodiments, the first area may, in part, overlap with thesecond area. For example, one or more detection elements may beassociated with the first area as well as the second area and/or one ormore detection elements of a first type may be associated with the firstarea while one or more detection elements of a second type may beassociated with the second area overlapping, at least in part, the firstarea. In other embodiments, the first area and the second area may bespatially separate from each other.

The one or more processors may, using the first and second signals,determine that one or more products have been placed in the first areawhile the second area includes at least one empty area. For example, ifthe detection elements include pressure sensors, the first signals mayinclude weight signals that match profiles of particular products (suchas the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include weight signals indicative of the absence ofproducts (e.g., by being equal to or within a threshold of a defaultvalue such as atmospheric pressure or the like). The disclosed weightsignals may be representative of actual weight values associated with aparticular product type or, alternatively, may be associated with arelative weight value sufficient to identify the product and/or toidentify the presence of a product. In some cases, the weight signal maybe suitable for verifying the presence of a product regardless ofwhether the signal is also sufficient for product identification. Inanother example, if the detection elements include light detectors (asdescribed above), the first signals may include light signals that matchprofiles of particular products (such as the mugs or plates depicted inthe example of FIG. 8A), and the second signals may include lightsignals indicative of the absence of products (e.g., by being equal toor within a threshold of a default value such as values corresponding toambient light or the like). For example, the first light signals may beindicative of ambient light being blocked by particular products, whilethe second light signals may be indicative of no product blocking theambient light. The disclosed light signals may be representative ofactual light patterns associated with a particular product type or,alternatively, may be associated with light patterns sufficient toidentify the product and/or to identify the presence of a product.

The one or more processors may similarly process signals from othertypes of sensors. For example, if the detection elements includeresistive or inductive sensors, the first signals may includeresistances, voltages, and/or currents that match profiles of particularproducts (such as the mugs or plates depicted in the example of FIG. 8Aor elements associated with the products, such as tags, etc.), and thesecond signals may include resistances, voltages, and/or currentsindicative of the absence of products (e.g., by being equal to or withina threshold of a default value such as atmospheric resistance, a defaultvoltage, a default current, corresponding to ambient light, or thelike). In another example, if the detection elements include acoustics,LIDAR, RADAR, or other reflective sensors, the first signals may includepatterns of returning waves (whether sound, visible light, infraredlight, radio, or the like) that match profiles of particular products(such as the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include patterns of returning waves (whether sound,visible light, infrared light, radio, or the like) indicative of theabsence of products (e.g., by being equal to or within a threshold of apattern associated with an empty shelf or the like).

Any of the profile matching described above may include direct matchingof a subject to a threshold. For example, direct matching may includetesting one or more measured values against the profile value(s) withina margin of error; mapping a received pattern onto a profile patternwith a residual having a maximum, minimum, integral, or the like withinthe margin of error; performing an autocorrelation, Fourier transform,convolution, or other operation on received measurements or a receivedpattern and comparing the resultant values or function against theprofile within a margin of error; or the like. Additionally oralternatively, profile matching may include fuzzy matching betweenmeasured values and/or patterns and a database of profiles such that aprofile with a highest level of confidence according to the fuzzysearch. Moreover, as depicted in the example of FIG. 8A, products, suchas product 803B, may be stacked and thus associated with a differentprofile when stacked than when standalone.

Any of the profile matching described above may include use of one ormore machine learning techniques. For example, one or more artificialneural networks, random forest models, or other models trained onmeasurements annotated with product identifiers may process themeasurements from the detection elements and identify productstherefrom. In such embodiments, the one or more models may useadditional or alternative input, such as images of the shelf (e.g., fromcapturing devices 125 of FIGS. 4A-4C explained above) or the like.

Based on detected products and/or empty spaces, determined using thefirst signals and second signals, the one or more processors maydetermine one or more aspects of planogram compliance. For example, theone or more processors may identify products and their locations on theshelves, determine quantities of products within particular areas (e.g.,identifying stacked or clustered products), identify facing directionsassociated with the products (e.g., whether a product is outward facing,inward facing, askew, or the like), or the like. Identification of theproducts may include identifying a product type (e.g., a bottle of soda,a loaf of broad, a notepad, or the like) and/or a product brand (e.g., aCoca-Cola® bottle instead of a Sprite® bottle, a Starbucks® coffeetumbler instead of a Tervis® coffee tumbler, or the like). Productfacing direction and/or orientation, for example, may be determinedbased on a detected orientation of an asymmetric shape of a product baseusing pressure sensitive pads, detected density of products, etc. Forexample, the product facing may be determined based on locations ofdetected product bases relative to certain areas of a shelf (e.g., alonga front edge of a shelf), etc. Product facing may also be determinedusing image sensors, light sensors, or any other sensor suitable fordetecting product orientation.

The one or more processors may generate one or more indicators of theone or more aspects of planogram compliance. For example, an indicatormay comprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct placement such as encoding intended coordinates of a product andactual coordinates on the shelf, with respect to product facingdirection and/or orientation such as encoding indicators of locationsthat have products not facing a correct direction and/or in an undesiredorientation, or the like.

In addition to or as an alternative to determining planogram compliance,the one or more processors may detect a change in measurements from oneor more detection elements. Such measurement changes may trigger aresponse. For example, a change of a first type may trigger capture ofat least one image of the shelf (e.g., using capturing devices 125 ofFIGS. 4A-4C explained above) while a detected change of a second typemay cause the at least one processor to forgo such capture. A first typeof change may, for example, indicate the moving of a product from onelocation on the shelf to another location such that planogram compliancemay be implicated. In such cases, it may be desired to capture an imageof the product rearrangement in order to assess or reassess productplanogram compliance. In another example, a first type of change mayindicate the removal of a product from the shelf, e.g., by an employeedue to damage, by a customer to purchase, or the like. On the otherhand, a second type of change may, for example, indicate the removal andreplacement of a product to the same (within a margin of error) locationon the shelf, e.g., by a customer to inspect the item. In cases whereproducts are removed from a shelf, but then replaced on the shelf (e.g.,within a particular time window), the system may forgo a new imagecapture, especially if the replaced product is detected in a locationsimilar to or the same as its recent, original position.

With reference to FIG. 8B and consistent with the present disclosure, astore shelf 850 may include a plurality of detection elements, e.g.,detection elements 851A and 851B. In the example of FIG. 8B, detectionelements 851A and 851B may comprise light sensors and/or other sensorsmeasuring one or more parameters (such as visual or otherelectromagnetic reflectance, visual or other electromagnetic emittance,or the like) based on electromagnetic waves from products, e.g., product853A and product 853B. Additionally or alternatively, as explained abovewith respect to FIG. 8B, detection elements 851A and 851B may comprisepressure sensors, other sensors measuring one or more parameters (suchas resistance, capacitance, or the like) based on physical contact (orlack thereof) with the products, and/or other sensors that measure oneor more parameters (such as current induction, magnetic induction,visual or other electromagnetic reflectance, visual or otherelectromagnetic emittance, or the like) based on physical proximity (orlack thereof) to products.

Moreover, although depicted as located on shelf 850, some detectionelements may be located next to shelf 850 (e.g., for magnetometers orthe like), across from shelf 850 (e.g., for image sensors or other lightsensors, light detection and ranging (LIDAR) sensors, radio detectionand ranging (RADAR) sensors, or the like), above shelf 850 (e.g., foracoustic sensors or the like), below shelf 850 (e.g., for pressuresensors, light detectors, or the like), or any other appropriate spatialarrangement. Further, although depicted as standalone in the example ofFIG. 8B, the plurality of detection elements may form part of a fabric(e.g., a smart fabric or the like), and the fabric may be positioned ona shelf to take measurements.

Detection elements associated with shelf 850 may be associated withdifferent areas of shelf 850, e.g., area 855A, area 855B, or the like.Although depicted as rows, areas 855A and 855B may comprise any areas ofshelf 850, whether contiguous (e.g., a square, a rectangular, or otherregular or irregular shape) or not (e.g., a plurality of rectangles orother regular and/or irregular shapes).

One or more processors (e.g., processing device 202) in communicationwith the detection elements (e.g., detection elements 851A and 851B) maydetect first signals associated with a first area and second signalsassociated with a second area. Any of the processing of the first andsecond signals described above with respect to FIG. 8A may similarly beperformed for the configuration of FIG. 8B.

In both FIGS. 8A and 8B, the detection elements may be integral to theshelf, part of a fabric or other surface configured for positioning onthe shelf, or the like. Power and/or data cables may form part of theshelf, the fabric, the surface, or be otherwise connected to thedetection elements. Additionally or alternatively, as depicted in FIGS.8A and 8B, individual sensors may be positioned on the shelf. Forexample, the power and/or data cables may be positioned under the shelfand connected through the shelf to the detection elements. In anotherexample, power and/or data may be transmitted wirelessly to thedetection elements (e.g., to wireless network interface controllersforming part of the detection elements). In yet another example, thedetection elements may include internal power sources (such as batteriesor fuel cells).

With reference to FIG. 9 and consistent with the present disclosure, thedetection elements described above with reference to FIGS. 8A and 8B maybe arranged on rows of the shelf in any appropriate configuration. Allof the arrangements of FIG. 9 are shown as a top-down view of a row(e.g., area 805A, area 805B, area 855A, area 855B, or the like) on theshelf. For example, arrangements 910 and 940 are both uniformdistributions of detection elements within a row. However, arrangement910 is also uniform throughout the depth of the row while arrangement940 is staggered. Both arrangements may provide signals that representproducts on the shelf in accordance with spatially uniform measurementlocations. As further shown in FIG. 9, arrangements 920, 930, 950, and960 cluster detection elements near the front (e.g., a facing portion)of the row. Arrangement 920 includes detection elements at a frontportion while arrangement 930 includes defection elements in a largerportion of the front of the shelf. Such arrangements may save power andprocessing cycles by having fewer detection elements on a back portionof the shelf. Arrangements 950 and 960 include some detection elementsin a back portion of the shelf but these elements are arranged lessdense than detection elements in the front. Such arrangements may allowfor detections in the back of the shelf (e.g., a need to restockproducts, a disruption to products in the back by a customer oremployee, or the like) while still using less power and fewer processingcycles than arrangements 910 and 940. And, such arrangements may includea higher density of detection elements in regions of the shelf (e.g., afront edge of the shelf) where product turnover rates may be higher thanin other regions (e.g., at areas deeper into a shelf), and/or in regionsof the shelf where planogram compliance is especially important.

FIG. 10A is a flow chart, illustrating an exemplary method 1000 formonitoring planogram compliance on a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1000may be used with any of the detection element arrays discussed abovewith reference to, for example, FIGS. 8A, 8B and 9. The order andarrangement of steps in method 1000 is provided for purposes ofillustration. As will be appreciated from this disclosure, modificationsmay be made to process 1000, for example, adding, combining, removing,and/or rearranging one or more steps of process 1000.

Method 1000 may include a step 1005 of receiving first signals from afirst subset of detection elements (e.g., detection elements 801A and801B of FIG. 8A) from among the plurality of detection elements afterone or more of a plurality of products (e.g., products 803A and 803B)are placed on at least one area of the store shelf associated with thefirst subset of detection elements. As explained above with respect toFIGS. 8A and 8B, the plurality of detection elements may be embeddedinto a fabric configured to be positioned on the store shelf.Additionally or alternatively, the plurality of detection elements maybe configured to be integrated with the store shelf. For example, anarray of pressure sensitive elements (or any other type of detector) maybe fabricated as part of the store shelf. In some examples, theplurality of detection elements may be configured to placed adjacent to(or located on) store shelves, as described above.

As described above with respect to arrangements 910 and 940 of FIG. 9,the plurality of detection elements may be substantially uniformlydistributed across the store shelf. Alternatively, as described abovewith respect to arrangements 920, 930, 950, and 960 of FIG. 9, theplurality of detection elements may be distributed relative to the storeshelf such that a first area of the store shelf has a higher density ofdetection elements than a second area of the store shelf. For example,the first area may comprise a front portion of the shelf, and the secondarea may comprise a back portion of the shelf.

In some embodiments, such as those including pressure sensors or othercontact sensors as depicted in the example of FIG. 8A, step 1005 mayinclude receiving the first signals from the first subset of detectionelements as the plurality of products are placed above the first subsetof detection elements. In some embodiments where the plurality ofdetection elements includes pressure detectors, the first signals may beindicative of pressure levels detected by pressure detectorscorresponding to the first subset of detection elements after one ormore of the plurality of products are placed on the at least one area ofthe store shelf associated with the first subset of detection elements.For example, the first signals may be indicative of pressure levelsdetected by pressure detectors corresponding to the first subset ofdetection elements after stocking at least one additional product abovea product previously positioned on the shelf, removal of a product fromthe shelf, or the like. In other embodiments where the plurality ofdetection elements includes light detectors, the first signals may beindicative of light measurements made with respect to one or more of theplurality of products placed on the at least one area of the store shelfassociated with the first subset of detection elements. Specifically,the first signals may be indicative of at least part of the ambientlight being blocked from reaching the light detectors by the one or moreof the plurality of products.

In embodiments including proximity sensors as depicted in the example ofFIG. 8B, step 1005 may include receiving the first signals from thefirst subset of detection elements as the plurality of products areplaced below the first subset of detection elements. In embodimentswhere the plurality of detection elements include proximity detectors,the first signals may be indicative of proximity measurements made withrespect to one or more of the plurality of products placed on the atleast one area of the store shelf associated with the first subset ofdetection elements.

Method 1000 may include step 1010 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A and 8B may be used for identification. Apattern associated with a product type may include a pattern (e.g., acontinuous ring, a discontinuous ring of a certain number of points, acertain shape, etc.) associated with a base of a single product. Thepattern associated with a product type may also be formed by a group ofproducts. For example, a six pack of soda cans may be associated with apattern including a 2×3 array of continuous rings associated with thesix cans of that product type. Additionally, a grouping of two literbottles may form a detectable pattern including an array (whetheruniform, irregular, or random) of discontinuous rings of pressurepoints, where the rings have a diameter associated with a particular2-liter product. Various other types of patterns may also be detected(e.g., patterns associated with different product types arrangedadjacent to one another, patterns associated with solid shapes (such asa rectangle of a boxed product), etc.). In another example, anartificial neural network configured to recognize product types may beused to analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine product types associated with products placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). In yet another example, a machinelearning algorithm trained using training examples to recognize producttypes may be used to analyze the signals received by step 1005 (such assignals from pressure sensors, from light detectors, from contactsensors, and so forth) to determine product types associated withproducts placed on an area of a shelf (such as an area of a shelfassociated with the first subset of detection elements).

In some embodiments, step 1010 may further include accessing a memorystoring data (e.g., memory device 226 of FIG. 2 and/or memory device 314of FIG. 3) associated with patterns of different types of products. Insuch embodiments, step 1010 may include using the first signals toidentify at least one product of a first type using a first pattern (ora first product model) and at least one product of a second type using asecond pattern (or a second product model). For example, the first typemay include one brand (such as Coca-Cola® or Folgers®) while the secondtype may include another brand (such as Pepsi® or Maxwell House 0). Inthis example, a size, shape, point spacing, weight, resistance or otherproperty of the first brand may be different from that of the secondbrand such that the detection elements may differentiate the brands.Such characteristics may also be used to differentiate like-branded, butdifferent products from one another (e.g., a 12-ounce can of Coca Cola,versus a 16 oz bottle of Coca Cola, versus a 2-liter bottle of CocaCola). For example, a soda may have a base detectable by a pressuresensitive pad as a continuous ring. Further, the can of soda may beassociated with a first weight signal having a value recognizable asassociated with such a product. A 16 ounce bottle of soda may beassociated with a base having four or five pressure points, which apressure sensitive pad may detect as arranged in a pattern associatedwith a diameter typical of such a product. The 16 ounce bottle of sodamay also be associated with a second weight signal having a value higherthan the weight signal associated with the 12 ounce can of soda. Furtherstill, a 2 liter bottle of soda may be associated with a base having aring, four or five pressure points, etc. that a pressure sensitive padmay detect as arranged in a pattern associated with a diameter typicalof such a product. The 2 liter bottle of soda may be associated with aweight signal having a value higher than the weight signal associatedwith the 12 ounce can of soda and 16 ounce bottle of soda.

In the example of FIG. 8B, the different bottoms of product 853A andproduct 853B may be used to differentiate the products from each other.For example, detection elements such as pressure sensitive pads may beused to detect a product base shape and size (e.g., ring, pattern ofpoints, asymmetric shape, base dimensions, etc.). Such a base shape andsize may be used (optionally, together with one or more weight signals)to identify a particular product. The signals may also be used toidentify and/or distinguish product types from one another. For example,a first type may include one category of product (such as soda cans)while a second type may include a different category of product (such asnotepads). In another example, detection elements such as lightdetectors may be used to detect a product based on a pattern of lightreadings indicative of a product blocking at least part of the ambientlight from reaching the light detectors. Such pattern of light readingsmay be used to identify product type and/or product category and/orproduct shape. For example, products of a first type may block a firstsubset of light frequencies of the ambient light from reaching the lightdetectors, while products of a second type may block a second subset oflight frequencies of the ambient light from reaching the light detectors(the first subset and second subset may differ). In this case the typeof the products may be determined based on the light frequenciesreaching the light detectors. In another example, products of a firsttype may have a first shape of shades and therefore may block ambientlight from reaching light detectors arranged in one shape, whileproducts of a second type may have a second shape of shades andtherefore may block ambient light from reaching light detectors arrangedin another shape. In this case the type of the products may bedetermined based on the shape of blocked ambient light. Any of thepattern matching techniques described above may be used for theidentification.

Additionally or alternatively, step 1010 may include using the at leastone pattern to determine a number of products placed on the at least onearea of the store shelf associated with the first subset of detectionelements. For example, any of the pattern matching techniques describedabove may be used to identify the presence of one or more product typesand then to determine the number of products of each product type (e.g.,by detecting a number of similarly sized and shaped product bases andoptionally by detecting weight signals associated with each detectedbase). In another example, an artificial neural network configured todetermine the number of products of selected product types may be usedto analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine the number of products of selected product typesplaced on an area of a shelf (such as an area of a shelf associated withthe first subset of detection elements). In yet another example, amachine learning algorithm trained using training examples to determinethe number of products of selected product types may be used to analyzethe signals received by step 1005 (such as signals from pressuresensors, from light detectors, from contact sensors, and so forth) todetermine the number of products of selected product types placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). Additionally or alternatively, step 1010may include extrapolating from a stored pattern associated with a singleproduct (or type of product) to determine the number of productsmatching the first signals. In such embodiments, step 1010 may furtherinclude determining, for example based on product dimension data storedin a memory, a number of additional products that can be placed on theat least one area of the store shelf associated with the second subsetof detection elements. For example, step 1010 may include extrapolatingbased on stored dimensions of each product and stored dimensions of theshelf area to determine an area and/or volume available for additionalproducts. Step 1010 may further include extrapolation of the number ofadditional products based on the stored dimensions of each product anddetermined available area and/or volume.

Method 1000 may include step 1015 of receiving second signals from asecond subset of detection elements (e.g., detection elements 851A and851B of FIG. 8B) from among the plurality of detection elements, thesecond signals being indicative of no products being placed on at leastone area of the store shelf associated with the second subset ofdetection elements. Using this information, method 1000 may include step1020 of using the second signals to determine at least one empty spaceon the store shelf. For example, any of the pattern matching techniquesdescribed above may be used to determine that the second signals includedefault values or other values indicative of a lack of product incertain areas associated with a retail store shelf. A default value maybe include, for example, a pressure signal associated with an un-loadedpressure sensor or pressure sensitive mat, indicating that no product islocated in a certain region of a shelf. In another example, a defaultvalue may include signals from light detectors corresponding to ambientlight, indicating that no product is located in a certain region of ashelf.

Method 1000 may include step 1025 of determining, based on the at leastone pattern associated with a detected product and the at least oneempty space, at least one aspect of planogram compliance. As explainedabove with respect to FIGS. 8A and 8B, the aspect of planogramcompliance may include the presence or absence of particular products(or brands), locations of products on the shelves, quantities ofproducts within particular areas (e.g., identifying stacked or clusteredproducts), facing directions associated with the products (e.g., whethera product is outward facing, inward facing, askew, or the like), or thelike. A planogram compliance determination may be made, for example, bydetermining a number of empty spaces on a shelf and determining alocation of the empty spaces on a shelf. The planogram determination mayalso include determining weight signal magnitudes associated withdetected products at the various detected non-empty locations. Thisinformation may be used by the one or more processors in determiningwhether a product facing specification has been satisfied (e.g., whethera front edge of a shelf has a suitable number of products or suitabledensity of products), whether a specified stacking density has beenachieved (e.g., by determining a pattern of detected products and weightsignals of the detected products to determine how many products arestacked at each location), whether a product density specification hasbeen achieved (e.g., by determining a ratio of empty locations toproduct-present locations), whether products of a selected product typeare located in a selected area of the shelf, whether all productslocated in a selected area of the shelf are of a selected product type,whether a selected number of products (or a selected number of productsof a selected product type) are located in a selected area of the shelf,whether products located in a selected area of a shelf are positioned ina selected orientation, or whether any other aspect of one or moreplanograms has been achieved.

For example, the at least one aspect may include product homogeneity,and step 1025 may further include counting occurrences where a productof the second type is placed on an area of the store shelf associatedwith the first type of product. For example, by accessing a memoryincluding base patterns (or any other type of pattern associated withproduct types, such as product models), the at least one processor maydetect different products and product types. A product of a first typemay be recognized based on a first pattern, and product of a second typemay be recognized based on a second, different pattern (optionally alsobased on weight signal information to aid in differentiating betweenproducts). Such information may be used, for example, to monitor whethera certain region of a shelf includes an appropriate or intended productor product type. Such information may also be useful in determiningwhether products or product types have been mixed (e.g., producthomogeneity). Regarding planogram compliance, detection of differentproducts and their relative locations on a shelf may aid in determiningwhether a product homogeneity value, ratio, etc. has been achieved. Forexample, the at least one processor may count occurrences where aproduct of a second type is placed on an area of the store shelfassociated with a product of a first type.

Additionally or alternatively, the at least one aspect of planogramcompliance may include a restocking rate, and step 1025 may furtherinclude determining the restocking rate based on a sensed rate at whichproducts are added to the at least one area of the store shelfassociated with the second subset of detection elements. Restocking ratemay be determined, for example, by monitoring a rate at which detectionelement signals change as products are added to a shelf (e.g., whenareas of a pressure sensitive pad change from a default value to aproduct-present value).

Additionally or alternatively, the at least one aspect of planogramcompliance may include product facing, and step 1025 may further includedetermining the product facing based on a number of products determinedto be placed on a selected area of the store shelf at a front of thestore shelf. Such product facing may be determined by determining anumber of products along a certain length of a front edge of a storeshelf and determining whether the number of products complies with, forexample, a specified density of products, a specified number ofproducts, and so forth.

Step 1025 may further include transmitting an indicator of the at leastone aspect of planogram compliance to a remote server. For example, asexplained above with respect to FIGS. 8A and 8B, the indicator maycomprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct (or brand) placement, product facing direction, or the like. Theremote server may include one or more computers associated with a retailstore (e.g., so planogram compliance may be determined on a local basiswithin a particular store), one or more computers associated with aretail store evaluation body (e.g., so planogram compliance may bedetermined across a plurality of retail stores), one or more computersassociated with a product manufacturer, one or more computers associatedwith a supplier (such as supplier 115), one or more computers associatedwith a market research entity (such as market research entity 110), etc.

Method 1000 may further include additional steps. For example, method1000 may include identifying a change in at least one characteristicassociated with one or more of the first signals (e.g., signals from afirst group or type of detection elements), and in response to theidentified change, triggering an acquisition of at least one image ofthe store shelf. The acquisition may be implemented by activating one ormore of capturing devices 125 of FIGS. 4A-4C, as explained above. Forexample, the change in at least one characteristic associated with oneor more of the first signals may be indicative of removal of at leastone product from a location associated with the at least one area of thestore shelf associated with the first subset of detection elements.Accordingly, method 1000 may include triggering the acquisition todetermine whether restocking, reorganizing, or other intervention isrequired, e.g., to improve planogram compliance. Thus, method 1000 mayinclude identifying a change in at least one characteristic associatedwith one or more of the first signals; and in response to the identifiedchange, trigger a product-related task for an employee of the retailstore.

Additionally or alternatively, method 1000 may be combined with method1050 of FIG. 10B, described below, such that step 1055 is performed anytime after step 1005.

FIG. 10B is a flow chart, illustrating an exemplary method 1050 fortriggering image capture of a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1050may be used in conjunction with any of the detection element arraysdiscussed above with reference to, for example, FIGS. 8A, 8B and 9. Theorder and arrangement of steps in method 1050 is provided for purposesof illustration. As will be appreciated from this disclosure,modifications may be made to process 1050, for example, adding,combining, removing, and/or rearranging one or more steps of process1050.

Method 1050 may include a step 1055 of determining a change in at leastone characteristic associated with one or more first signals. Forexample, the first signals may have been captured as part of method 1000of FIG. 10A, described above. For example, the first signals may includepressure readings when the plurality of detection elements includespressure sensors, contact information when the plurality of detectionelements includes contact sensors, light readings when the plurality ofdetection elements includes light detectors (for example, from lightdetectors configured to be placed adjacent to (or located on) a surfaceof a store shelf configured to hold products, as described above), andso forth.

Method 1050 may include step 1060 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A, 8B, and step 1010 may be used foridentification.

Method 1050 may include step 1065 of determining a type of eventassociated with the change. For example, a type of event may include aproduct removal, a product placement, movement of a product, or thelike.

Method 1050 may include step 1070 of triggering an acquisition of atleast one image of the store shelf when the change is associated with afirst event type. For example, a first event type may include removal ofa product, moving of a product, or the like, such that the first eventtype may trigger a product-related task for an employee of the retailstore depending on analysis of the at least one image. The acquisitionmay be implemented by activating one or more of capturing devices 125 ofFIGS. 4A-4C, as explained above. In some examples, the triggeredacquisition may include an activation of at least one projector (such asprojector 632). In some examples, the triggered acquisition may includeacquisition of color images, depth images, stereo images, active stereoimages, time of flight images, LIDAR images, RADAR images, and so forth.

Method 1050 may include a step (not shown) of forgoing the acquisitionof at least one image of the store shelf when the change is associatedwith a second event type. For example, a second event type may includereplacement of a removed product by a customer, stocking of a shelf byan employee, or the like. As another example, a second event type mayinclude removal, placement, or movement of a product that is detectedwithin a margin of error of the detection elements and/or detectedwithin a threshold (e.g., removal of only one or two products; movementof a product by less than 5 cm, 20 cm, or the like; moving of a facingdirection by less than 10 degrees; or the like), such that no imageacquisition is required.

FIGS. 11A-11E illustrate example outputs based on data automaticallyderived from machine processing and analysis of images captured inretail store 105 according to disclosed embodiments. FIG. 11Aillustrates an optional output for market research entity 110. FIG. 11Billustrates an optional output for supplier 115. FIGS. 11C and 11Dillustrate optional outputs for employees of retail store 105. And FIG.11E illustrates optional outputs for user 120.

FIG. 11A illustrates an example graphical user interface (GUI) 500 foroutput device 145A, representative of a GUI that may be used by marketresearch entity 110. Consistent with the present disclosure, marketresearch entity 110 may assist supplier 115 and other stakeholders inidentifying emerging trends, launching new products, and/or developingmerchandising and distribution plans across a large number of retailstores 105. By doing so, market research entity 110 may assist supplier115 in growing product presence and maximizing or increasing new productsales. As mentioned above, market research entity 110 may be separatedfrom or part of supplier 115. To successfully launch a new product,supplier 115 may use information about what really happens in retailstore 105. For example, supplier 115 may want to monitor how marketingplans are being executed and to learn what other competitors are doingrelative to certain products or product types. Embodiments of thepresent disclosure may allow market research entity 110 and suppliers115 to continuously monitor product-related activities at retail stores105 (e.g., using system 100 to generate various metrics or informationbased on automated analysis of actual, timely images acquired from theretail stores). For example, in some embodiments, market research entity110 may track how quickly or at what rate new products are introduced toretail store shelves, identify new products introduced by variousentities, assess a supplier's brand presence across different retailstores 105, among many other potential metrics.

In some embodiments, server 135 may provide market research entity 110with information including shelf organization, analysis of skewproductivity trends, and various reports aggregating information onproducts appearing across large numbers of retail stores 105. Forexample, as shown in FIG. 11A, GUI 1100 may include a first display area1102 for showing a percentage of promotion campaign compliance indifferent retail stores 105. GUI 1100 may also include a second displayarea 1104 showing a graph illustrating sales of a certain productrelative to the percentage of out of shelf. GUI 1100 may also include athird display area 1106 showing actual measurements of different factorsrelative to target goals (e.g., planogram compliance, restocking rate,price compliance, and other metrics). The provided information mayenable market research entity 110 to give supplier 115 informed shelvingrecommendations and fine-tune promotional strategies according toin-store marketing trends, to provide store managers with a comparisonof store performances in comparison to a group of retail stores 105 orindustry wide performances, and so forth.

FIG. 11B illustrates an example GUI 1110 for output device 145B used bysupplier 115. Consistent with the present disclosure, server 135 may usedata derived from images captured in a plurality of retail stores 105 torecommend a planogram, which often determines sales success of differentproducts. Using various analytics and planogram productivity measures,server 135 may help supplier 115 to determine an effective planogramwith assurances that most if not all retail stores 105 can execute theplan. For example, the determined planogram may increase the probabilitythat inventory is available for each retail store 105 and may bedesigned to decrease costs or to keep costs within a budget (such asinventory costs, restocking costs, shelf space costs, etc.). Server 135may also provide pricing recommendations based on the goals of supplier115 and other factors. In other words, server 135 may help supplier 115understand how much room to reserve for different products and how tomake them available for favorable sales and profit impact (for example,by choosing the size of the shelf dedicated to a selected product, thelocation of the shelf, the height of the shelf, the neighboringproducts, and so forth). In addition, server 135 may monitor nearreal-time data from retail stores 105 to determine or confirm thatretail stores 105 are compliant with the determined planogram ofsupplier 115. As used herein, the term “near real-time data,” in thecontext of this disclosure, refers to data acquired or generated, etc.,based on sensor readings and other inputs (such as data from imagesensors, audio sensors, pressure sensors, checkout stations, etc.) fromretail store 105 received by system 100 within a predefined period oftime (such as time periods having durations of less than a second, lessthan a minute, less than an hour, less than a day, less than a week,etc.).

In some embodiments, server 135 may generate reports that summarizeperformance of the current assortment and the planogram compliance.These reports may advise supplier 115 of the category and the itemperformance based on individual SKU, sub segments of the category,vendor, and region. In addition, server 135 may provide suggestions orinformation upon which decisions may be made regarding how or when toremove markdowns and when to replace underperforming products. Forexample, as shown in FIG. 11B, GUI 1110 may include a first display area1112 for showing different scores of supplier 115 relative to scoresassociated with its competitors. GUI 1110 may also include a seconddisplay area 1114 showing the market share of each competitor. GUI 1110may also include a third display area 1116 showing retail measurementsand distribution of brands. GUI 1110 may also include a fourth displayarea 1118 showing a suggested planogram. The provided information mayhelp supplier 115 to select preferred planograms based on projected orobserved profitability, etc., and to ensure that retail stores 105 arefollowing the determined planogram.

FIGS. 11C and 11D illustrate example GUIs for output devices 145C, whichmay be used by employees of retail store 105. FIG. 11C depicts a GUI1120 for a manager of retail store 105 designed for a desktop computer,and FIG. 11D depicts GUI 1130 and 1140 for store staff designed for ahandheld device. In-store execution is one of the challenges retailstores 105 have in creating a positive customer experience. Typicalin-store execution may involve dealing with ongoing service events, suchas a cleaning event, a restocking event, a rearrangement event, andmore. In some embodiments, system 100 may improve in-store execution byproviding adequate visibility to ensure that the right products arelocated at preferred locations on the shelf. For example, using nearreal-time data (e.g., captured images of store shelves) server 135 maygenerate customized online reports. Store managers and regionalmanagers, as well as other stakeholders, may access custom dashboardsand online reports to see how in-store conditions (such as, planogramcompliance, promotion compliance, price compliance, etc.) are affectingsales. This way, system 100 may enable managers of retail stores 105 tostay on top of burning issues across the floor and assign employees toaddress issues that may negatively impact the customer experience.

In some embodiments, server 135 may cause real-time automated alertswhen products are out of shelf (or near out of shelf), when pricing isinaccurate, when intended promotions are absent, and/or when there areissues with planogram compliance, among others. In the example shown inFIG. 11C, GUI 1120 may include a first display area 1122 for showing theaverage scores (for certain metrics) of a specific retail store 105 overa selected period of time. GUI 1120 may also include a second displayarea 1124 for showing a map of the specific retail store 105 withreal-time indications of selected in-store execution events that requireattention, and a third display area 1126 for showing a list of theselected in-store execution events that require attention. In anotherexample, shown in FIG. 11D, GUI 1130 may include a first display area1132 for showing a list of notifications or text messages indicatingselected in-store execution events that require attention. Thenotifications or text messages may include a link to an image (or theimage itself) of the specific aisle with the in-store execution event.In another example, shown in FIG. 11D, GUI 1140 may include a firstdisplay area 1142 for showing a display of a video stream captured byoutput device 145C (e.g., a real-time display or a near real-timedisplay) with augmented markings indicting a status of planogramcompliance for each product (e.g., correct place, misplaced, not inplanogram, empty, and so forth). GUI 1140 may also include a seconddisplay area 1144 for showing a summary of the planogram compliance forall the products identified in the video stream captured by outputdevice 145C. Consistent with the present disclosure, server 135 maygenerate within minutes actionable tasks to improve store execution.These tasks may help employees of retail store 105 to quickly addresssituations that can negatively impact revenue and customer experience inthe retail store 105.

FIG. 11E illustrates an example GUI 1150 for output device 145D used byan online customer of retail store 105. Traditional online shoppingsystems present online customers with a list of products. Productsselected for purchase may be placed into a virtual shopping cart untilthe customers complete their virtual shopping trip. Virtual shoppingcarts may be examined at any time, and their contents can be edited ordeleted. However, common problems of traditional online shopping systemsarise when the list of products on the website does not correspond withthe actual products on the shelf. For example, an online customer mayorder a favorite cookie brand without knowing that the cookie brand isout-of-stock. Consistent with some embodiments, system 100 may use imagedata acquired by capturing devices 125 to provide the online customerwith a near real-time display of the retail store and a list of theactual products on the shelf based on near real-time data. In oneembodiment, server 135 may select images without occlusions in the fieldof view (e.g., without other customers, carts, etc.) for the nearreal-time display. In one embodiment, server 135 may blur or erasedepictions of customers and other people from the near real-timedisplay. As used herein, the term “near real-time display,” in thecontext of this disclosure, refers to image data captured in retailstore 105 that was obtained by system 100 within a predefined period oftime (such as less than a second, less than a minute, less than about 30minutes, less than an hour, less than 3 hours, or less than 12 hours)from the time the image data was captured.

Consistent with the present disclosure, the near real-time display ofretail store 105 may be presented to the online customer in a mannerenabling easy virtual navigation in retail store 105. For example, asshown in FIG. 11E, GUI 1150 may include a first display area 1152 forshowing the near real-time display and a second display area 1154 forshowing a product list including products identified in the nearreal-time display. In some embodiments, first display area 1152 mayinclude different GUI features (e.g., tabs 1156) associated withdifferent locations or departments of retail store 105. By selectingeach of the GUI features, the online customer can virtually jump todifferent locations or departments in retail store 105. For example,upon selecting the “bakery” tab, GUI 1150 may present a near real-timedisplay of the bakery of retail store 105. In addition, first displayarea 1152 may include one or more navigational features (e.g., arrows1158A and 1158B) for enabling the online customer to virtually movewithin a selected department and/or virtually walk through retail store105. Server 135 may be configured to update the near real-time displayand the product list upon determining that the online customer wants tovirtually move within retail store 105. For example, after identifying aselection of arrow 1158B, server 135 may present a different section ofthe dairy department and may update the product list accordingly. Inanother example, server 135 may update the near-real time display andthe product list in response to new captured images and new informationreceived from retail store 105. Using GUI 1150, the online customer mayhave the closest shopping experience without actually being in retailstore 105. For example, an online customer can visit the vegetabledepartment and decide not to buy tomatoes after seeing that they are notripe enough.

Monitoring Product Shortages Over Time

In the retail world, stores tend to avoid technology for detecting anon-shelf product shortage due to the large number of unnecessary orirrelevant alerts these systems often generate. Many of these alerts arein response to shortages that are part of the “usual flow” of the store,which need no intervention on the part of the store. Other alerts,however, may relate to real shortages, but the large number of alertsmay desensitize the store to the alerts and, as a result, these realshortages tend to be ignored without any intervention. Disclosed systemsprovide alerts when there is a real shortage of products from a certainproduct type. These alerts may be based on time periods associated withobserved shortages (e.g., shortages that persist beyond an acceptabletime limit). The system, however, may avoid providing unnecessaryalerts, for example, by analyzing data from a number of sources suchthat actual shortages generate alerts, rather than normal shortages thatoccur during the hour-by-hour operation of the store.

FIG. 12 illustrates an exemplary embodiment of a memory device 1200containing software modules consistent with the present disclosure. Asshown, memory device 1200 may include a retail store communicationmodule 1202, a data processing module 1204, an external sourcescommunication module 1206, a shift module 1208, an alert module 1210, adatabase access module 1212, and a database 1214. Modules 1202, 1204,1206, 1208, 1210, and 1212 may contain software instructions forexecution by at least one processor (e.g., processing device 202)associated with system 100. Retail store communication module 1202, dataprocessing module 1204, external sources communication module 1206,shift data module 1208, alert module 1210, database access module 1212,and database 1214 may cooperate to perform multiple operations. Forexample, retail store communication module 1202 may receive image datafrom a plurality of image capturing devices 125 in retail store 105,data processing module 1204 may use the received image data to determineinformation about a displayed inventory of products of a certain producttype on at least one store shelf, external sources communication module1206 may obtain scheduling data about additional product inventoryscheduled to arrive to the retail store (e.g., from supplier 115), shiftdata module 1208 may use employee scheduling data to determineinformation about scheduled restocking of the certain product type, andalert module 1210 may generate one or more alerts and/or suggestionsregarding a product shortage of products in retail store 105, forexample, alerting management to the shortage of a particular product.

In some embodiments, memory device 1200 may be part of system 100, forexample, memory device 226. Alternatively, memory device 1200 may bestored in an external database or an external storage communicativelycoupled with server 135, such as one or more databases or memoriesaccessible over communication network 150. Further, in otherembodiments, the components of memory device 1200 may be distributed inmore than one server and more than one memory device.

In some embodiments, retail store communication module 1202 may receiveinformation from sensors located in retail store 105. In one example,retail store communication module 1202 may receive images captured by aplurality of image sensors fixedly mounted in retail store 105 orreceive data derived from images captured by plurality of image sensorsfixedly mounted in retail store 105. In another example, retail storecommunication module 1202 may receive image data (e.g., images or dataderived from images) from robotic capturing devices configured tonavigate autonomously within retail store 105 and to capture images ofmultiple types of products on at least one store shelf. In yet anotherexample, retail store communication module 1202 may receive data fromone or more shelf sensors configured to measure quantities of productsplaced on a store shelf. The one or more shelf sensors may includepressure sensitive pads, light detectors, weight sensors, light sensors,odor sensors, resistive sensors, ultrasonic sensors, and more.

In some embodiments, data processing module 1204 may process theinformation collected by retail store communication module 1202 todetermine a quantity of products located on at least one shelf in retailstore 105. In some embodiments, products of the same product type may belocated in a display area accessible to customers, while another portionof the products found in retail store 105 may be located in storage areanot accessible to customers. The term “storage area” broadly includesall storage areas of products not displayed for sale in the retailstore, for example, high shelves, below display areas of shelving unit,in a dedicated storage area, and more. In one embodiment, dataprocessing module 1204 may determine product quantities for multipletypes of products at a number of time intervals based on image datareceived from a plurality of image sensors fixedly mounted in retailstore 105 (e.g., as illustrated in FIG. 4A). In another embodiment, dataprocessing module 1204 may determine quantities of multiple types ofproducts using a combination of image data and data from one or moreshelf sensors configured to measure properties of products placed on astore shelf (e.g., as illustrated in FIG. 8A). For example, the dataprocessing module 1204 may analyze the data received from detectionelements attached to store shelves, alone or in combination with imagescaptured in retail store 105, to determine the quantity of a certainproduct on at least one store shelf. In another embodiment, dataprocessing module 1204 may determine whether a product is out-of-stockusing a combination of image data and information about productsavailable in at least one storage area of retail store 105. For example,the data processing module 1204 may use inventory records and image datato determine information about the current inventory of a certain typeof product. The current inventory of the product may include theinventory displayed for sale and the inventory stored in retail store105.

In addition, data processing module 1204 may determine the quantity ofproducts of a certain product type by analyzing image data and othertypes of captured data. For example, data processing module 1204 maycalculate a shortage duration or estimated duration of a productshortage. In some cases, determining the quantity of products of acertain product type on at least one store shelf may include aggregatingdata from two or more locations in retail store 105. For example, in oneembodiment, data processing module 1204 may monitor the quantity of acertain product type displayed on a first shelving unit, on a secondshelving unit, and/or in a storing area. For example, the first shelvingunit may be located in a specific aisle, the second shelving unit may belocated in a display next to the cashier, and the storing area may be ina cold room.

In some embodiments, external sources communication module 1206 mayobtain scheduling data about additional products scheduled to arrive tothe retail store. In one embodiment, external sources communicationmodule 1206 may receive the scheduling data from one or more of entitiesin the supply chain. In one example, the scheduling data may be receivedfrom online services (e.g., from a server that store data on shipmentsorders). In another example, the scheduling data may be received fromsupplier 115 associated with the products (e.g., from a supplierscheduling deliveries). In another example, the scheduling data may bereceived from a market research entity 110 (e.g., statistics aboutdemand for certain products). In another example, the scheduling datamay be received from a shipment company that delivers the products(e.g., from an IoT sensor in a cargo ship). In another example, thescheduling data may be received from a distribution company thatdelivers the products (e.g., from an agent who supplies the products toretail stores). In one embodiment, external sources communication module1206 may determine the scheduling data about the additional productsusing information collected from one or more of entities in the supplychain and other data sources, for example, Enterprise Resource Planning(ERP), Warehouse Management Software (WMS), and Supply Chain Management(SCM) applications. In addition, external sources communication module1206 may generate the scheduling data using analytics of data associatedwith past delivery of products. For example, the external sourcescommunication module 1206 may determine that a shipment of dairyproducts is scheduled to be delivered to retail store 105 based on pastdeliveries.

In some embodiments, shift data module 1208 may process informationassociated with employees in retail store 105. For example, shift datamodule 1208 may receive information from one or more systems (e.g.,employee time-clock, weekly employee schedules, daily tasks assigned toemployees) and determine whether an employee is currently assigned torestock a shelf or whether to generate an alert to have a product thatis available in storage be restocked on a store shelf.

In some embodiments, alert module 1210 may use the output of dataprocessing module 1204 and shift data module 1208 to determine whetheror not to generate an alert for a product shortage in the retail store.In some embodiments, an alert may include a recommendation to mitigatethe shortage, for example, the alert may generate an assignment to anemployee to restock the particular product. In other embodiments, alertmodule 1210 may determine not to generate an alert, thereby, notalerting management to false shortages or desensitizing management byover-alerting to shortages. For example, in response to a shortageidentified based on image data, alert module 1210 may receive shift datafrom shift data module 1208 indicating an employee is scheduled torestock. Because a restock of the product is already scheduled, alertmodule 1210 will not generate a shortage alert.

In some embodiments, database access module 1212 may cooperate withdatabase 1214 to retrieve stored scheduling data such as product supplyinformation. The product supply information may include, for example, aschedule of arrivals of additional products, inventory records, checkoutdata, calendar data, and historical product turnover data. Dataprocessing module 1204 and shift data analysis module 1208 may use theproduct supply information stored in database 1214 to determine one ormore factors related to product deliveries, product turnover, and/orrestock frequency. Database 1214 may include separate databases,including, for example, a vector database, raster database, tiledatabase, viewport database, and/or a user input database, configured tostore data. The data stored in database 1214 may be received frommodules 1202-1212, server 135, from any communication device associatedwith retail stores 105, market research entity 110, suppliers 115, andusers 120. Moreover, the data stored in database 1814 may be provided asinput using data entry, data transfer, or data uploading.

Modules 1202-1212 may be implemented in software, hardware, firmware, amix of any of those, or the like. For example, if the modules areimplemented in software, the modules may be stored in a server (e.g.,server 135) or distributed over a plurality of servers. In someembodiments, any one or more of modules 1202-1212 and data associatedwith database 1214 may be stored in database 140 and/or located onserver 135, which may include one or more processing devices. Processingdevices of server 135 may be configured to execute the instructions ofmodules 1202-1212. In some embodiments, aspects of modules 1202-1212 mayinclude software, hardware, or firmware instructions (or a combinationthereof) executable by one or more processors, alone, or in variouscombinations with each other. For example, modules 1202-1212 may beconfigured to interact with each other and/or other modules of server135 to perform functions consistent with disclosed embodiments.

Reference is now made to FIGS. 13A-13C, which illustrate shortagescenarios at a number of discrete times (e.g., a first time of scene1300 of FIG. 13A, a second time of scene 1312 of FIG. 13B, and a thirdtime of scene 1314 of FIG. 13C) that may be detected by an image capturedevice 1302 (e.g., image capture device 125) of system 100 (e.g.,processing device 202). For example, image capture device 1302 maycapture one or more images of a store shelf 1304 and product types 1306,1308, and 1310. In some embodiments, a processor (e.g., processingdevice 202) of system 100 may access information describing a desiredplacement of products of a certain product type (e.g., product type1306, 1308, or 1310) on at least one store shelf (e.g., shelf 1304).Processing device 202 may receive, at data processing module 1204, imagedata from a plurality of image sensors (e.g., image capture device 1302)including a plurality of images captured at different times anddepicting the at least one store shelf 1304. Processing device 202 mayaccess, e.g., via data interface 1212, information describing a desiredplacement of products of a certain product type on at least one storeshelf. The information describing desired placement of objects mayinclude a planogram, a realogram, other kind of instructions, rules,obligations, etc.

Data processing module 1204 may analyze, from among the plurality ofimages, a first image captured at a first time to determine that a firstquantity of products of the certain product type at the first timeexceeds a quantity threshold associated with the information describingthe desired placement of products. In some embodiments, alert module1210 may receive, e.g., from data access module 1212, a quantitythreshold associated with each product type. For example, data accessmodule 1212 may query database 1214 to retrieve a quantity threshold foreach of product types 1306, 1308, and 1310 identified in images of shelf1304.

In some embodiments, the quantity threshold may be selected based ontype of product (e.g., brand, category, particular product, etc.). Inother embodiments, the quantity threshold may be based on the locationof shelf within the store. For example, the same product may have onethreshold for a promotional display and a second threshold for a shelfwithin the category area. In other embodiments, the quantity thresholdmay be based on the type of shelf (e.g., an end display, a display at aregister, or a shelf within the category area). The quantity thresholdmay also be based on time of day, retail store, adjunct products, numberof facings of the product type, and so forth. As an example, a smallstore may have different thresholds than a large store, a store with afirst number of store associates may have different thresholds than astore with fewer associates, and a discount store may have differentthreshold than a high end store, etc. In another example, a product mayhave a higher quantity threshold at peak times than at off-peak times.

In some embodiments, the threshold quantity, as well as otherpredetermine values discussed below, may be determined based oninformation including at least one of: a schedule of arrivals of theadditional products (e.g., dates, times, updates from a supplier, etc.),known orders of the additional products (e.g., quantity of scheduledproducts), delivery records (e.g., the reports on the delivery progressof the additional products and/or data predicting future delivery ofadditional products), cost data (e.g., the different costs associatedwith the additional products), calendar data (e.g., holidays, sportgames, etc.), historical product turnover data (e.g., the amount ofavocados sold per last April), average turnover (e.g., per hour in aday, per day in a week, per special events), and more. In oneembodiment, the information may be obtained from multiple sources (e.g.,market research entity 110, supplier 115, and multiple retail stores105) and may include different types of parameters. In some embodiments,the obtained information may be stored digitally in memory device 226,stored in database 140, received using network interface 206 throughcommunication network 150, received from a user, determined byprocessing device 202, and so forth.

In some embodiments, at least one product type may be a fresh product.The processing device may analyze the plurality of images to determine aquality of the products on the at least one store shelf, and todetermine if there is a product shortage based on both the quantity anddetermined quality of the products on the at least one store shelf, asdescribed in further detail with reference to FIGS. 18-25. As anexample, even if the quantity of tomatoes on a shelf is greater than thethreshold quantity, a shortage may still be reported if their quality islow.

FIG. 13B depicts a scene 1312 at a second time, the second time beinglater than the first time. Data processing module 1204 may analyze, fromamong the plurality of images, a second image captured at a second timelater than the first time to determine that a second quantity ofproducts of the certain product type at the second time is below thequantity threshold associated with the information describing thedesired placement of products. Alert module 1210 may receive, e.g., fromdata access module 1212, a quantity threshold associated with eachproduct type and/or information describing the desired placement ofproducts.

For example, in scene 1312, data processing module 1204 may determinethat the shelf 1304, at the second time, is stocked with seven ofproduct type 1306 (fewer than at the first time), two of product type1308 (fewer than at the first time), and six of product type 1310 (thesame quantity as at the first time). In this example, the respectivequantity thresholds for each of product type 1306, 1308, and 1310 is twounits, five units, and three units, respectively. Also, in this example,the quantity of product type 1308 is below the threshold amount of fiveunits. Although the quantity of product type 1306 on shelf 1304 hasdecreased, the quantity is not less than the threshold of two units. Insome embodiments, the quantity threshold may be a product density,number of units, etc.

For each of the three product types 1306, 1308, and 1310 at a first timein scenario 1300, based on the determination that the first quantity ofproducts of the certain product type at the first time exceeds thequantity threshold, alert module 1210 may avoid initiating anunscheduled action to cause restocking the at least one store shelf withproducts of the certain product type at least until a third time laterthan the second time. For example, because the shelf 1304 is stockedwith a number of units for each product that is greater than thequantity threshold for each product, no alert is generated by thesystem.

FIG. 13C depicts a scene 1314 at a third time, the third time beinglater than the first time and the second time. Data processing module1204 may analyze, from among the plurality of images, a third imagecaptured at the third time to determine that a third quantity ofproducts of the certain product type at the third time is also below thequantity threshold associated with the information describing thedesired placement of products. Again, alert module 1210 may receive,e.g., from data access module 1212, a quantity threshold associated witheach product type and/or information describing the desired placement ofproducts.

Referring to FIG. 13C, at the third time shown in scenario 1314, shelf1304 is now stocked with six of product type 1306 (fewer than at thefirst time and at the second time), two of product type 1308 (fewer thanat the first time, but the same amount as at the second time), and fiveof product type 1310 (fewer than at the first time and at the secondtime). Remembering that the quantity thresholds for product types 1306,1308, and 1310 are two units, five units, and three units, respectively,the quantity of product 1308 is below its quantity threshold at both thesecond time (scenario 1312) and third time (scenario 1314).

Thus, for product type 1308, based on the determination that the secondquantity of products of the certain product type at the second time andthe third quantity of products at the third time are below the quantitythreshold, alert module 1210 may provide information indicative of aproduct shortage at the at least one store shelf. In this example, eventhough the number of products of product types 1306 and 1310 havedecreased between the first, second, and third times, neither is belowthe threshold quantity and, therefore, alert module 1210 will notgenerate a shortage alert for product types 1306 and 1310.

Alert module 1210 may provide an alert, information regarding a detectedshortage, and/or suggestion of how to mitigate a shortage to an entityassociated with retail store 105. Consistent with the presentdisclosure, the entity associated with retail store 105 may be anemployee (human or robot) of retail store 105, a customer of retailstore 105, a user of system 100 that monitors retail store 105 (e.g.,market research entity 110), a processing device that control operationsassociated with retail store 105 (e.g., ERP system, WMS system, and SCMsystem), and any other device or individual capable of implementingsuggestions regarding restocking products in retail store 105.

In some embodiments, the processing device may provide the informationindicative of a product shortage to an external system configured toaggregate product shortage information from a plurality of retailstores, for example, via external communication module 1206. Theprocessing device may receive, from the external system, informationrelating to the aggregation of product shortage information from theplurality of retail stores. Such information may include, for example,the estimated time span until restocking, type of product (brand,category, particular product, etc.), location within the store, time inday associated with the product shortage, information about the retailstore (size, type, location, etc.), information about adjunct products,information about the planogram (number of facings, etc.). For example,the aggregated data may be used to generate statistics about shortagedurations associated with the certain product type. These statistics maybe used by retail store 105, market research entity 110, and/or supplier115 to inform operational decisions and evaluate performance.

FIG. 14A depicts an exemplary method 1400 for processing images capturedin a retail store and automatically identifying a product shortage. Forpurposes of illustration, in the following description, reference ismade to certain components of system 100. It will be appreciated,however, that other implementations are possible and that othercomponents may be utilized to implement the exemplary method. It willalso be readily appreciated that the illustrated method can be alteredto modify the order of steps, delete steps, or further includeadditional steps.

At step 1402, a processing device (e.g., processing device 202) mayaccess information describing a desired placement of products of acertain product type on at least one store shelf in retail store 105.For example, processing device may retrieve placement information, viadatabase access module 1212, from database 1214. In other embodiments,placement information may be received from a market research entity 110or a supplier 115 via external sources communication module 1206. Theplacement information may include a planogram, a realogram,instructions, rules, obligations, etc. For example, the productplacement information may reflect an agreed upon arrangement or productdensity as described in a contract between retail store 105 and supplier115. In another example, a market research entity 110 may provide retailstore 105, or supplier 115, with an optimized product placement planbased on aggregate data analysis of the product type at a number ofretail stores. In yet another example, the realogram may be determinedaccording to actual placement of products on the at least one storeshelf, for example, as determined by analyzing images of the at leastone store shelf captured over time, as determined from manual reports,and so forth.

At step 1404, the processing device may receive image data from aplurality of image sensors mounted in the retail store, the image dataincluding a plurality of images captured at different times anddepicting the at least one store shelf in retail store 105. In oneembodiment, the image data may be generated from the plurality of imagesacquired by any one of image capturing devices 125 illustrated in FIG.4A-4C. For example, the set of images may be acquired by a plurality ofimage sensors fixedly mounted in retail store 105. Details of the imagecapturing devices mounted in retail store 105 and configured to acquirethe plurality of images are described with references to FIGS. 5A-5C.The plurality of images may be used by data processing module 1204 toderive image data such as processed images (e.g., cropped images, videostreams, and more) or information derived from the captured images(e.g., data about products identified in captured images, data that maybe used to construct a 3D image, and more). In one embodiment, theprocessing device may periodically receive a plurality of images thatcorresponds with the certain product type on the at least one shelf. Forexample, the processing device may receive an image of the plurality ofimages at predetermined time intervals (e.g., every minute, every 5minutes, every 15 minutes, every 30 minutes, etc.). In anotherembodiment, the processing device may receive an image of the pluralityof images in response to a detection of a certain identified event. Inone example, the processing device may receive the plurality of imagesafter an event, such as a detected lifting of a product from theshelving unit.

At step 1406, the processing device may analyze, from among theplurality of images, a first image captured at a first time to determinethat a first quantity of products of the certain product type at thefirst time exceeds a quantity threshold associated with the informationdescribing the desired placement of products. In one embodiment, theprocessing device may determine quantity of products using a combinationof image data derived from the received plurality of images and datafrom one or more additional sensors. In another embodiment, theprocessing device may determine a quantity of products using data fromthe one or more additional sensors without using the image data. Forexample, the one or more additional sensors may include weight sensors,pressure sensors, touch sensors, light sensors, and detection elementsas described in relation to FIGS. 8A, 8B and 9, and so forth. Suchsensors may be used alone or in combination with image capturing devices125. For example, the processing device may analyze the data receivedfrom weight sensors alone or in combination with images captured fromthe retail store, to estimate the number of units of product on a storeshelving unit.

In some embodiments, analyzing an image of the plurality of images todetermine a quantity of products may further comprise preprocessing theimage data. In some examples, the image data may be preprocessed (e.g.,by data processing module 1204) by transforming the image data using atransformation function to obtain a transformed image data, and thepreprocessed image data may include the transformed image data. Forexample, the transformation function may comprise convolutions, visualfilters (such as low-pass filters, high-pass filters, band-pass filters,all-pass filters, etc.), nonlinear functions, and so forth. In someexamples, the image data may be preprocessed by smoothing the imagedata, for example, by using Gaussian convolution, using a median filter,and so forth. In some examples, the image data may be preprocessed toobtain a different representation of the image data. For example, thepreprocessed image data may include: a representation of at least partof the image data in a frequency domain; a Discrete Fourier Transform ofat least part of the image data; a Discrete Wavelet Transform of atleast part of the image data; a time/frequency representation of atleast part of the image data; a representation of at least part of theimage data in a lower dimension; a lossy compression representation ofat least part of the image data; a lossless compression representationof at least part of the image data; a time order series of any of theabove; any combination of the above; and so forth. In some examples, theimage data may be preprocessed to extract edges, and the preprocessedimage data may include information based on and/or related to theextracted edges. In some examples, the image data may be preprocessed toextract visual features from the image data. Some examples of suchvisual features include edges, corners, blobs, ridges, Scale InvariantFeature Transform (SIFT) features, temporal features, and so forth. Oneof ordinary skill in the art will recognize that the image data may bepreprocessed using other kinds of preprocessing methods.

In some embodiments, analyzing an image of the plurality of images todetermine a quantity of products may further comprise using a trainedmachine learning model to analyze the image and/or the preprocessedimages to determine the quantity of products. For example, the machinelearning model may be trained using training examples to determinenumbers of products of selected product types from images and/or videos.An example of such training example may include an image together with anumber of products of the selected product types depicted in the image.In some embodiments, analyzing an image of the plurality of images todetermine a quantity of products may further comprise using anartificial neural network (such as a deep neural network, aconvolutional neural network, etc.) to analyze the image and/or thepreprocessed image to determine the quantity of products. For example,the artificial neural network may be configured to determine numbers ofproducts of selected product types from images and/or videos.

At step 1408, the processing device may analyze, from among theplurality of images, a second image captured at a second time later thanthe first time to determine that a second quantity of products of thecertain product type at the second time is below the quantity thresholdassociated with the information describing the desired placement ofproducts. Analysis of the second image may include preprocessing, or theprocessing techniques discussed above with reference to step 1406. As instep 1406, the second quantity may be determined, based at least inpart, on information collected from one or more other types of sensors,e.g., a weight sensor.

At step 1410, based on the determination that the first quantity ofproducts of the certain product type at the first time exceeds thequantity threshold, the processor may avoid initiating an unscheduledaction to cause restocking the at least one store shelf with products ofthe certain product type at least until a third time later than thesecond time. For example, alert module 1210 may receive the data fromanalysis of the first and second images at steps 1406 and determine thatthe quantity of the product exceeds a quantity threshold for thatproduct. In some embodiments, the quantity threshold may be based on aweight of the product on the shelf, as detected by a weight sensor.

At step 1412, the processor may analyze, from among the plurality ofimages, a third image captured at the third time to determine that athird quantity of products of the certain product type at the third timeis also below the quantity threshold associated with the informationdescribing the desired placement of products. As described above, theanalysis of the third image may include one or more preprocessing steps,and/or may include analyzing data from a number of different types ofsensors.

At step 1414, based on the determination that the second quantity ofproducts of the certain product type at the second time and the thirdquantity of products at the third time are below the quantity threshold,the processor may provide information indicative of a product shortageat the at least one store shelf. In some embodiments, the informationindicative of a shortage may include one or more suggestions formitigating the shortage. Some examples of such suggestions may include asuggestion to restock the at least one store shelf, a suggestion forreordering products on the at least one store shelf, a suggestion formoving products of the certain product type from another part of theretail store to the at least one store shelf, a suggestion to allocatespace of the at least one store shelf to another product type (forexample when products of the certain product type are unavailable forrestocking), and so forth.

Consistent with the present disclosure, the processing device mayperform big data analytics on the information about the displayedinventory and/or the information about the product. Typically, the bigdata analytics may be executed in an automatic or autonomous mannerwithout using any user input during the learning analysis. In oneembodiment, the processing device may perform large-scale data analysison the received information image data and scheduling data to discover,detect, or learn new data. For example, the large-scale data analysisinvolves the process of examining large quantities of data (e.g., datamining, data discovery, etc.) to extract new or previously unknowninteresting data or patterns, such as unusual demand events for acertain type of product or an average shortage duration. The processingdevice may also perform large-scale data analysis on the stored data(e.g., machine learning analysis, data modeling, pattern recognition,predictive analysis, correlation analysis, etc.) to predict, calculate,or identify implicit relationships or inferences within the stored data.For example, the processing device may identify at least one patterncorrelating a combination of different parameters to product sales. Thedifferent parameters may include the quantity of products on display,the number of employees working at the time of a product shortage, thepredicted demand for the products, the location at which the productsare being displayed, the time of day, the day of the week, the price ofthe product, and the like. The identified at least one pattern may beused for determining whether or not to generate an alert indicating aproduct shortage or to generate suggestions of how to mitigate anongoing shortage in retail store 105.

FIG. 14B depicts a flowchart of an example process 1416 executed by aprocessing device of system 100 (e.g., processing device 202) forprocessing images captured in a retail store and automaticallyidentifying a product shortage in retail store 105. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 100. It will be appreciated, however, that otherimplementations are possible and that other components may be used toimplement example process 1416. It will also be readily appreciated thatthe example process 1416 can be altered to modify the order of steps,delete steps, or further include additional steps.

Process 1416 may begin when the processing device analyzes a receivedset of images to determine that the second quantity and the thirdquantity are both below the quantity threshold (block 1418). Forexample, block 1418 may occur after, or in tandem, with steps 1406-1414of method 1400.

The processing device may access information indicating that a scheduledrestocking event related to the certain product type is about to takeplace within a period of time after the third time. For example, shiftdata module 1208 may analyze current workstreams and/or employee data todetermine whether a restocking event is scheduled or has been assignedto an employee. The processing device may determine whether thescheduled restocking event falls within a period of time after the thirdtime (decision block 1420).

The time threshold may be selected based on type of product (brand,category, particular product, etc.). In some embodiments, the timethreshold may be based on the location of a shelf within the store orthe type of shelf. For example, the same product may have one timethreshold for a promotional display and a second time threshold for ashelf within the category area. The time threshold may be based on thetime of day such that, for example, a shorter time threshold indicateshigher urgency at peak times. The time threshold may also depend on thetype or size of the retail store. For example, a small store may havedifferent time threshold than a large store, a store with a first numberof store associates may have different time threshold than a store withfewer associates, or a discount store may have different time thresholdthan a high end store, etc. In other embodiments the time threshold maybe based on adjunct products, number of facings of the product type, andthe like.

The processing device may withhold providing the information indicativeof a product shortage when the period of time is less than the timethreshold (block 1422). For example, if a scheduled restocking event isabout to take place, there is no need to provide an alert to a storemanager or employee. When the period of time is greater than a timethreshold, the processing device may provide the information indicativeof a product shortage (block 1424), for example, via an alert providedto a device of a store manager. In some embodiments, if the time isgreater than the time threshold, process 1416 may proceed to block 1426.

In some embodiments, the processing device may access informationindicative of a quantity of products of the certain product typeavailable in a storage area of the retail store (block 1426). As usedherein, storage area may refer to any area of retail store 105 storingproduct that is not currently being offered for sale. For example, astorage area may include a warehouse, industrial freeze or refrigerator,stock room, etc. Information indicative of quantity of productsavailable in the storage area may include inventory data maintained indatabase 1214, image data collected by image capture devices 125 andanalyzed by data processing module 1204, and/or other inventoryinformation. For example, the available quantity may be based onpurchase order data, sales data, real-time NFC information (e.g., RFIDtag scans), and/or optical information (e.g., barcode or QR code scans).

Based on the determined product availability information (decision block1428), the system may determine whether or not to indicate a productshortage and provide shortage information to one or more systems ordevices. If there are no available products of the certain product typeavailable in the storage area of the retail store, the processing devicemay withhold providing the information indicative of a product shortage(block 1430). For example, if a product is out-of-stock, or there is lowor no inventory in a storage area, the processing device may not reporta shortage of the product, since no mitigating action (e.g., restockingthe shelf) may be taken.

In some embodiments, the processing device may cause a change to thedesired placement of products when there are no available products ofthe certain product type in the storage area of the retail store. Forexample, if a product is out-of-stock, alert module 1210 may generateinformation indicative of a change in the desired placement of productsto avoid empty shelves or to move remaining product to a more heavilytrafficked area of the store. In some embodiments, alert module 1210 maygenerate updated information describing a desired placement of productsof a certain product type on at least one store shelf, e.g., a modifiedplanogram.

As an example, a product shortage may be identified for Product A, whichis displayed on a shelf and at a register. If the shortage occurs at theregister display and there is no inventory of Product A in the store'sstorage area, alert module 1210 may generate instructions to move someof Product A on the shelf to the register display. In some embodiments,the product shortage information may include information indicative of aquantity of products in each location. In another example, instructionsto restock one location may be assigned a lower priority if the customercan find the item in a second location. The instructions may beprovided, for example, to an employee or manager via a user device. Insome embodiments, the instructions change the product display may bebased on instructions from market research entity 110, supplier 115, oron other historical or aggregate product data.

When there are available at least some products of the certain producttype in the storage area of the retail store, the processing device mayprovide the information indicative of a product shortage, for example,to a device associated with a store manager or employee (block 1426). Insome embodiments, when there are available at least some products of thecertain product type in the storage area of the retail store, theprocessing device may communicate with one or more systems of the retailstore 105 to cause a restocking event related to the product type whenthere are available products of the certain product type available inthe storage area of the retail store.

In some embodiments, when there are available at least some products ofthe certain product type in the storage area of the retail store,process 1416 may proceed to block 1432. The processing device may accessinformation indicative of pending assignments of one or more storeemployees, for example, via shift module 1208. Pending assignments mayinclude, for example, employee-specific assignments, scheduled events,shift start and end times, etc. The processing device may then determinewhether or not a pending assignment exists in one or more systems of theretail store 105 for restocking the at least one store shelf (decisionblock 1434). When there is a pending assignment for restocking the atleast one store shelf, the processing device may withhold providing theinformation indicative of a product shortage when there is a pendingassignment for restocking the at least one store shelf (block 1422). Forexample, the processing device will not generate extraneous shortagealerts if the product shortage is known and being handled or if anemployee has already begun or is assigned to begin restocking the atleast one shelf. In some embodiments, the pending assignment, e.g., torestock a shelf in response to a shortage, may be created manually by astore associate, by a customer (via a crowd sourcing application), byanother image processing system, in response to data from other sensors,etc.

When there is no pending assignment for restocking the at least onestore shelf, process 1416 may provide the information indicative of aproduct shortage (block 1424). In some embodiments, when there is nopending assignment for restocking the at least one store shelf, process1416 may proceed to block 1436.

The processing device may also determine a number of times within acertain time period that the information indicative of a productshortage at the at least one store shelf was previously provided (block1436). The number of times may include shortage information inputmanually by a store associate, by a customer (via a crowd sourcingapplication) or may include automatically generated shortage reportsfrom another image processing system, in response to data from othersensors, etc. The system may then compare the number of times with apredetermined threshold (decision block 1438).

The processing device may, for example, withhold providing theinformation indicative of a product shortage when the number of times isat least equal to the predefined number (block 1440), for example, inorder to reduce number of alerts that are ignored by the storeassociates (e.g., the alerts may be ignored by the store associatessince no remedy to the shortage is currently available). Thepredetermined number may be selected based on type of product (brand,category, particular product, etc.), for example, by selecting a firstpredetermined number for a first product type and selecting a secondpredetermined number for a second product type. In some embodiments, thepredetermined number may be based on the location of a shelf within thestore or the type of shelf. For example, there may be a number of timesfor a promotional display in a high-traffic area and a second number fora shelf within the category area. The number may be based on the time ofday such that, for example, a lower number indicates higher urgency atpeak times. The number may also depend on the type or size of the retailstore. For example, a small store may have a different predeterminednumber than a large store, a store with a first number of storeassociates may have a different predetermined number than a store withfewer associates, or a discount store may have a different number oftimes than a high end store, etc. In other embodiments the predeterminednumber may be based on adjunct products, number of facings of theproduct type, and the like.

In some embodiments, the system may initiate a restocking event. Forexample, the processing device may initiate a restocking event if ashortage has been reported a number of times greater than or equal tothe predetermined number and no restocking event is scheduled, nor isany employee assigned a restocking task.

It is to be understood that the steps of process 1416 may occur in anyorder, and may be executed simultaneously or in tandem.

When the number of times is smaller than a predetermined number theprocessing device may provide the information indicative of a productshortage (block 1424). At block 1424, the information may be providedvia one or more user devices, or may be provided to one or more otherautomated systems associated with the retail store 105, market researchentity 110, or supplier 115. For example, the information may beprovided as described above in connection with step 1414.

FIG. 14C is a flowchart of an example of a process 1442 for processingimages captured in a retail store and automatically identifying aproduct shortage. Process 1442 may begin when the processing deviceanalyzes a received set of images to determine that the second quantityand the third quantity are both below the quantity threshold and mayoccur after, or in tandem, with steps 1406-1414 of method 1400 and steps1418-1440 of process 1416.

As previously described, the processing device may receive a pluralityof images taken by an image capture device (e.g., image capture device125) at times T1, T2, Ti, T3, T4, and T5. Also as previously described,the processing device may analyze, from among the plurality of images,each image captured at each time to determine a quantity of products(Q1, Q2, Qi, Q3, Q4 and Q5, respectively) of the certain product type ateach time. As described above, the analysis of each image may includeone or more preprocessing steps, and/or may include analyzing data froma number of different types of sensors.

In some embodiments, the processing device may analyze, from among theplurality of images, an additional image captured at an intermediatetime (Ti) between the second time (T2) and the third time (T3) todetermine a fourth, or intermediate, quantity (Qi) of products of thecertain product type at the intermediate time. The processing device maydetermine whether the quantity of product at T3 is below a quantitythreshold (Qthresh) (decision block 1444) and may also determine whetherthe intermediate quantity is below the threshold quantity (decisionblock 1446).

If the third quantity and intermediate quantity are both greater thanthe threshold quantity, the processor may withhold providing theinformation indicative of a product shortage (block 1446). If the thirdquantity is less than the threshold quantity, but the intermediatequantity is greater than the threshold quantity, the processor may alsowithhold providing the information indicative of a product shortage(block 1446). However, if both the third quantity and the intermediatequantity are both less than the threshold quantity, the processingdevice may provide the information indicative of a product shortage ormay continue with one or more steps of process 1416 (block 1448).

For example, if the third quantity is below the threshold, but anintermediate quantity is above the threshold, the difference in quantitymay be due to normal fluctuations. As an example, product on the atleast one shelf may be depleted and restocked several times during peakhours. Even if both Q2 and Q3 are below the threshold, if Qi is greaterthan the threshold, the shelf may have been recently restocked at a timebetween T2 and T3. In another example, the processing device may receiveimage data captured at a number of intermediate times between T2 and T3.

In some embodiments, the threshold quantity for the intermediate imagesmay be different from the threshold quantity at times T2 and T3. Forexample, the threshold for the intermediate images may be based on thequantity in the image at T2 such that the threshold represents anyincrease in the number of products on the at least one shelf between T2and Ti. In some embodiments, the interim threshold may be a number ofproducts, a product density, a percentage of the quantity of products atT2, etc. Thus, the intermediate images may be used to account for noise,thereby reducing erroneous shortage alerts. Noise may occur due to, forexample, detection errors by the computer vision, a customer returning aproduct, etc.

In some embodiments, the processing device may determine whether anelapsed time between the second time and the third time is less than orgreater than a predetermined shortage time duration (block 1450). Theduration may be selected based on type of product (brand, category,particular product, etc.). In some embodiments, the duration may bebased on the location of a shelf within the store or the type of shelf.For example, there may be one duration for a promotional display in ahigh-traffic area and another duration for a shelf within the categoryarea. The duration may be based on the time of day such that, forexample, a shorter duration indicates higher urgency at peak times. Theduration may also depend on the type or size of the retail store. Forexample, a small store may have a different duration than a large store,a store with a first number of store associates may have differentduration than a store with fewer associates, or a discount store mayhave a different duration than a high end store, etc. In otherembodiments the duration may be based on adjunct products, number offacings of the product type, and the like.

Thus, if the difference between T3 and T2 is less than the predeterminedduration, the processing device may withhold providing the informationindicative of a product shortage based on the determination. Forexample, the duration may be one minute. A change in product quantityover a period of time of less than a minute would not trigger a shortagealert. Similarly, in some embodiments, if the difference between T3 andT2 is on the order of, for example, days, a shortage alert is nottriggered. In some embodiments, if the elapsed time between the secondtime and the third time is greater than the predetermined shortage timeduration, the at least one processor is further configured to provide arecommendation to place products from a differing product type on the atleast one store shelf (block 1452). As an example, if a product has notbeen restocked for ten days (e.g., T2 occurred ten days prior to T3 andQ2 and Q3 are both less than the quantity threshold), the system shouldnot generate continuous shortage alerts. Rather, such a scenario mayoccur due to other factors, e.g., stocking changes, contractual changeswith a vendor, seasonal items, etc.

If the elapsed time between the second time and the third time is lessthan the predetermined shortage time duration, the at least oneprocessor may be further configured to trigger capturing of anadditional image to determine a quantity of products of the certainproduct type at a time after the third time, e.g., at T4 (block 1454).As an example, assuming that the selected shortage duration is between 4hours and 8 hours, and the elapsed time is 3.5 hours, the system maytrigger one or more image capture devices to capture an additional imageafter 0.5 hours. Thus, system receives an image at regular intervalssuch that a shortage in the appropriate time frame may be detected.

In some embodiments, the processing device may analyze a fourth imagecaptured at a fourth time (T4) later than the third time and a fifthimage captured at a fifth time (T5) later than the fourth time todetermine that a restocking event occurred between the fourth time andthe fifth time (block 1456). The processing device may estimate a timespan until restocking based on at least one of the first time and thesecond time and based on at least one of the fourth time and the fifthtime and provide information based on the estimated time span untilrestocking (block 1458). For example, the estimated time span may beused in calculating future shortage durations, determining image captureintervals, providing metrics on anticipated shortages, etc.

It is to be understood that the steps of process 1416 may occur in anyorder, and may be executed simultaneously or in tandem.

Aggregating Product Shortage Information

Product shortages may cause inefficiencies and losses for retail storesdue to inefficient use of space, decreased product turnover, inabilityto meet customer demand, etc. Disclosed embodiments may reduce productshortage durations in retail stores by using aggregated product shortageinformation from many stores and applying big data algorithms toidentify patterns and/or common factors contributing to the shortage ofa product.

FIG. 15 illustrates an exemplary embodiment of a memory device 1500containing software modules consistent with the present disclosure. Inparticular, as shown, memory device 1500 may include a retail storecommunication module 1502, a data processing module 1504, an externalsources communication module 1506, a data analysis module 1508, anaction determination module 1510, a database access module 1512, and adatabase 1514. Modules 1502, 1504, 1506, 1508, 1510, and 1512 maycontain software instructions for execution by at least one processor(e.g., processing device 202) associated with system 100. Retail storecommunication module 1502, data processing module 1504, external sourcescommunication module 1506, data analysis module 1508, actiondetermination module 1510, database access module 1512, and database1514 may cooperate to perform multiple operations. For example, retailstore communication module 1502 may receive image data from a pluralityof image capturing devices 125 in retail store 105, data processingmodule 1504 may use the received image data to determine informationabout a displayed inventory of a plurality of products, external sourcescommunication module 1506 may obtain scheduling data about additionalproducts scheduled to arrive to the retail store (e.g., from supplier115), data analysis module 1508 may use data from a number of sources(e.g., from database 1514, image processing module 1504, one or moreimage capture devices 125, one or more other sensors, etc.) to identifyone or more patterns and/or factors associated with product shortages atone or more retail stores, and action determination module 1510 maydetermine at least one suggested action for addressing a shortage ofproducts in retail store 105 based on the identified patterns andfactors, for example, changing product prices, restocking shelves,changing the number of employees staffed during a shift, etc.

In some embodiments, memory device 1500 may be part of system 100, forexample, memory device 226. Alternatively, memory device 1500 may bestored in an external database or an external storage communicativelycoupled with server 135, such as one or more databases or memoriesaccessible over communication network 150. Further, in otherembodiments, the components of memory device 1500 may be distributed inmore than one server and more than one memory device.

In some embodiments, retail store communication module 1502 may receiveinformation from sensors located in retail store 105. In one example,retail store communication module 1802 may receive images captured by aplurality of image sensors fixedly mounted in retail store 105 orreceive data derived from images captured by plurality of image sensorsfixedly mounted in retail store 105. In another example, retail storecommunication module 1502 may receive image data (e.g., images or dataderived from images) from robotic capturing devices configured tonavigate autonomously within retail store 105 and to capture images ofmultiple types of products. In yet another example, retail storecommunication module 1502 may receive data from one or more shelfsensors configured to measure properties of products placed on a storeshelf. The one or more shelf sensors may include pressure sensitivepads, light detectors, weight sensors, light sensors, odor sensors,resistive sensors, ultrasonic sensors, and more.

Retail store communication module 1502 may also receive informationassociated with the management and staffing of retail store 105. Forexample, retail store communication module 1502 may receive employeetimecards, shift information, scheduled tasks (e.g., restocking events),assigned tasks, etc. Data analysis module 1508 may use this data,coupled with information from data processing module 1504, to identifyone or more patterns associated with product shortages. For example, ahigher number of shortages may occur at a time in which fewer restockingevents are scheduled.

In some embodiments, data processing module 1504 may process theinformation collected by retail store communication module 1502 todetermine and monitor one or more product shortages (e.g., as describedwith reference to FIGS. 12, 13A-13C, and 14A-14C). A portion of theproducts found in retail store 105 may be located in a display areaaccessible to customers, while another portion of the products found inretail store 105 may be located in storage area not accessible tocustomers. The term “storage area” broadly includes all storage areas ofproducts not displayed for sale in the retail store, for example, highshelves, below display areas of shelving unit, in a dedicated storagearea, and more. In one embodiment, image processing module 1504 mayidentify one or more shortages for multiple types of products based onimage data received from a plurality of image sensors fixedly mounted inretail store 105 (e.g., as illustrated in FIG. 4A). In anotherembodiment, data processing module 1504 may identify a shortage ofproducts using a combination of image data and data from one or moreshelf sensors configured to measure properties of products placed on astore shelf (e.g., as illustrated in FIG. 8A). For example, the imageprocessing module 1504 may analyze the data received from detectionelements attached to store shelves, alone or in combination with imagescaptured in retail store 105, to identify and monitor shortages ofproducts. In another embodiment, data processing module 1504 mayidentify a product shortage or product shortage information using acombination of image data and information about products available in atleast one storage area of retail store 105. For example, the dataprocessing module 1504 may use inventory records and image data todetermine information about the current inventory of multiple types ofproducts. The current inventory may include the inventory displayed forsale and the inventory stored in retail store 105.

In addition, data processing module 1504 may also analyze textualinformation derived from image data, analyze visual appearance of theproducts, calculate a duration of products on a shelf, determiningturnover of products from the product type on the shelf, determine ashortage duration, and determining storage requirements associated withthe products.

In some embodiments, data processing module 1504 may preprocess receivedimage data by transforming the image data using a transformationfunction to obtain a transformed image data. Preprocessed image data mayinclude the transformed image data. For example, the transformationfunction may comprise convolutions, visual filters (such as low-passfilters, high-pass filters, band-pass filters, all-pass filters, etc.),nonlinear functions, and so forth. In some examples, the image data maybe preprocessed by smoothing the image data, for example, by usingGaussian convolution, using a median filter, and so forth. In someexamples, the image data may be preprocessed to obtain a differentrepresentation of the image data. For example, the preprocessed imagedata may include: a representation of at least part of the image data ina frequency domain; a Discrete Fourier Transform of at least part of theimage data; a Discrete Wavelet Transform of at least part of the imagedata; a time/frequency representation of at least part of the imagedata; a representation of at least part of the image data in a lowerdimension; a lossy compression representation of at least part of theimage data; a lossless compression representation of at least part ofthe image data; a time order series of any of the above; any combinationof the above; and so forth. In some examples, the image data may bepreprocessed to extract edges, and the preprocessed image data mayinclude information based on and/or related to the extracted edges. Insome examples, the image data may be preprocessed to extract visualfeatures from the image data. Some examples of such visual featuresinclude edges, corners, blobs, ridges, Scale Invariant Feature Transform(SIFT) features, temporal features, and so forth. One of ordinary skillin the art will recognize that the image data may be preprocessed usingother kinds of preprocessing methods.

In some embodiments, external sources communication module 1506 mayobtain scheduling data about additional products scheduled to arrive tothe retail store. In one embodiment, external sources communicationmodule 1506 may receive the scheduling data from one or more of entitiesin the supply chain. In one example, the scheduling data may be receivedfrom online services (e.g., from a server that store data on shipmentsorders). In another example, the scheduling data may be received fromsupplier 115 associated with the products (e.g., from a farmer ormanufacturer that produced the products). In another example, thescheduling data may be received from a market research entity 110 (e.g.,statistics about demand for certain products). In another example, thescheduling data may be received from a shipment company that deliversthe products (e.g., from an IOT sensor in a cargo ship). In anotherexample, the scheduling data may be received from a distribution companythat delivers the products (e.g., from an agent who supplies theproducts to retail stores). In one embodiment, external sourcescommunication module 1506 may determine the scheduling data about theadditional products using information collected from one or more ofentities in the supply chain and other data sources, for example,Enterprise Resource Planning (ERP), Warehouse Management Software (WMS),and Supply Chain Management (SCM) applications. In addition, externalsources communication module 1506 may generate the scheduling data usinganalytics of data associated with past delivery of products. Forexample, the external sources communication module 1506 may determinethat a shipment is scheduled to be delivered to retail store 105 basedon past deliveries.

In some embodiments, data analysis module 1508 may process theinformation collected by external sources communication module 1506 andgenerated by image processing module 1504 to determine one or morepatterns associated with one or more product shortages in retail store105. In some embodiments, data analysis module 1508 may receiveinformation from data processing module 1504 associated with a number ofretail stores. Data analysis module 1508 may aggregate data from two ormore retail stores. The aggregated data may be sourced from retailstores based on, for example, store size, geographic location, number ofemployees, store price-point, etc. In other embodiments, data analysismodule 1508 may analyze data from a single store to determine patternsand factors associated with product shortages.

Consistent with the present disclosure, data analysis module 1508 mayuse artificial neural networks, convolutional neural networks, machinelearning models, image regression models, and other processingtechniques to determine shortage patterns and/or factors. For example,data analysis module 1508 may use a regression model on the receivedemployment or delivery data to predict future product shortages inretail store 105. In some embodiments, data processing module 1504 mayanalyze the image data to detect a plurality of occurrences of productshortages of multiple product types in at least one retail store. Basedon the analysis, data analysis module 1508 may identify a patternassociated with the product shortages of the multiple product types andprovide information associated with the identified pattern to theentity. For example, the identified pattern and/or identified commonfactors associated with one or more shortages may be displayed to a uservia a communication device. The device may display information regardingthe shortages, for example, that a particular department has a higher orlower rate of shortages than another department. This patterninformation may aid a retail store manager in evaluating which action touse to prevent future shortages.

In some embodiments, action determination module 1510 may use the outputof data processing module 1504 and data analysis module 1508 todetermine at least one suggestion regarding an action for mitigating ashortage or a predicted shortage. A suggested action may be, forexample, changing the location of products, the price of products, therestocking rates of products, a promotion assigned to one or moreproducts, and more. Generally, the purpose of the action is to preventshortages or to reduce the duration of shortages. Examples ofsuggestions generated by action determination module 1510 areillustrated in FIG. 17A and described in detail below.

As an example, data processing module 1508 may receive inventory datafrom the plurality of retail stores. The inventory data may be received,e.g., via retail store communication module 1502. The received inventorydata may include details about a chain-of-supply of products of at leastone product type associated with times in which product shortages of theat least one product type occur. Data processing module 1508 mayidentify the at least one common factor based on analysis of the imagedata and the inventory data. Exemplary common factors may includedelivery times associated with each shortage, space in storage duringeach shortage, etc.

Action suggestion module 1510 may receive the common factor informationand identify an action for reducing future shortages. For example, theaction for potentially reducing product shortage durations of futureshortages of the at least one product type may include changing anelement in the chain-of-supply of products of the at least one producttype. Changing an element may include increase the amount of productdelivered each time, increasing the frequency of deliveries, generatinga new purchase order or contract with the updated delivery information,etc.

In another example, data analysis module 1508 may receive employmentdata from a plurality of retail stores. The employment data may includedetails about store employees that worked during shifts in which aproduct shortage of at least one product type occurred. Data analysismodule 1508 may identify at least one common factor based on analysis ofthe image data and the employment data. For example, a common factor maybe the number of employees during shift, the rank or position of theemployees during a shift, the training of an employee during the shift,etc. In some embodiments, a common factor may be the position, training,and/or rank of an employee assigned to one or more tasks. Based on thecommon factors identified by data analysis module 1508, actionsuggestion module 1510 may identify an action for potentially reducingproduct shortage durations of future shortages of the at least oneproduct type. The identified action may include changing employmentdynamics in future shifts associated with shifts in which a productshortage of at least one product type has occurred. For example, theaction may be to hire more people between for a particular shift,decrease or increase shift lengths, redistribute employee workloads,etc.

In another example, data analysis module 1508 may receive restockingdata from a plurality of retail stores, including, for example, detailsabout restocking practices of retail stores in which a product shortageof at least one product type has occurred. Data analysis module 1508 mayidentify the at least one common factor based on analysis of the imagedata and the restocking data. For example, the product shortage may beassociated with an hour-by-hour restocking schedule or may be associatedwith an on-demand restocking schedule. Action suggestion module 1510 mayidentify an action for potentially reducing product shortage durationsof future shortages of the at least one product type. The identifiedaction may include changing restocking practices of retail stores inwhich a product shortage of at least one product type has occurred. Forexample, the identified action may be to switch from a restock rate ofone restock per day, to increase the rate to two restocking events perday.

The action determination module 1510 may use retail store communicationmodule 1502, external sources communication module 1806, or both toprovide the at least one suggested action to an entity associated withretail store 105, market research entity 110, and/or supplier 115. Inone example, action determination module 1510 may determine that anincreased restock rate for a particular shift may prevent a predictedshortage and provide the recommended restock rate and shift to aprocessing device associated with retail store 105. In a second example,action determination module 1510 may determine that a particular retailstore typically experiences regular shortages of a product and maytransmit a suggestion to a processing device associated with a specificsupplier to deliver a larger quantity of the product in futuredeliveries.

In some embodiments, database access module 1512 may cooperate withdatabase 1514 to retrieve stored scheduling data such as product supplyinformation. The product supply information may include, for example, aschedule of arrivals of additional products, inventory records, checkoutdata, calendar data, and historical product turnover data. Dataprocessing module 1504 and data analysis module 1508 may use the productsupply information stored in database 1514 to determine one or morefactors or patterns associated with detected shortages. Database 1514may include separate databases, including, for example, a vectordatabase, raster database, tile database, viewport database, and/or auser input database, configured to store data. The data stored indatabase 1514 may be received from modules 1502-1512, server 135, fromany communication device associated with retail stores 105, marketresearch entity 110, suppliers 115, and users 120. Moreover, the datastored in database 1514 may be provided as input using data entry, datatransfer, or data uploading.

Modules 1502-1512 may be implemented in software, hardware, firmware, amix of any of those, or the like. For example, if the modules areimplemented in software, the modules may be stored in a server (e.g.,server 135) or distributed over a plurality of servers. In someembodiments, any one or more of modules 1502-1512 and data associatedwith database 1514 may be stored in database 140 and/or located onserver 135, which may include one or more processing devices. Processingdevices of server 135 may be configured to execute the instructions ofmodules 1502-1512. In some embodiments, aspects of modules 1502-1512 mayinclude software, hardware, or firmware instructions (or a combinationthereof) executable by one or more processors, alone, or in variouscombinations with each other. For example, modules 1502-1512 may beconfigured to interact with each other and/or other modules of server135 to perform functions consistent with disclosed embodiments.

FIG. 16 depicts an exemplary method 1600 for reducing product shortagedurations in retail stores based on analysis of image data. For purposesof illustration, in the following description, reference is made tocertain components of system 100. It will be appreciated, however, thatother implementations are possible and that other components may be usedto implement the exemplary method. It will also be readily appreciatedthat the illustrated method can be altered to modify the order of steps,delete steps, or further include additional steps.

Method 1600 may be executed, for example, by a system having acommunication interface configured to receive image data from aplurality of retail stores (e.g., retail store communication module1502). In step 1602, image data may be received from a plurality ofretail stores. The image data received by step 1602 (for example, from aparticular retail store, from the plurality of retail stores, etc.) maybe indicative of a product shortage of at least one product typerelative to information describing a desired placement of products on atleast one store shelf. The image data may include, for example, imagesor data derived from image processing (e.g., by data processing module1504). The received image data may be indicative of a product shortageof at least one product type relative to information describing adesired placement of products of the at least one product type on atleast one store shelf. The information describing desired placement ofobjects may include a planogram, a realogram, other kind ofinstructions, rules, obligations, etc.

In one embodiment, the images and/or image data may be acquired by anyone of image capturing devices 125 illustrated in FIG. 4A-4C. Forexample, the images may be acquired by a plurality of image sensorsfixedly mounted in retail store 105. Details of the image capturingdevices mounted in retail store 105 and configured to acquire images aredescribed with references to FIGS. 5A-5C. The images may be used toderive image data such as processed images (e.g., cropped images, videostreams, images preprocessed as described above, and more) orinformation derived from the captured images (e.g., data about productsidentified in captured images, data that may be used to construct a 3Dimage, and more). In one embodiment, the processing device mayperiodically receive images that correspond with the current inventoryof the products on the shelf in retail store 105. In some embodiments,the processing device may receive images and/or image data atpredetermined time intervals (e.g., every minute, every 5 minutes, every15 minutes, every 30 minutes, etc.). In another embodiment, theprocessing device may receive the images and/or image data in responseto a detection of a certain identified event (e.g., a shortage).

At step 1604, a processing device (e.g., processing device 202) mayanalyze the image data to detect a plurality of occurrences of productshortages of at least one product type in the plurality of retail storesand determine product shortage durations associated with the pluralityof occurrences. For example, product shortages may be identified basedon analysis of the quantity of product on a shelf at subsequent times.In some embodiments, the at least one product type is considered to bein a state of shortage when a number of products of the at least oneproduct type on the at least one store shelf is below a predefined timeduration threshold. For example, a product is in shortage when there isa 30% decrease in product quantity for a predetermined amount of time.In other embodiments, the shortage may be based on a decrease in productweight or decrease in product density. The predefined time durationthreshold may be determined based on the at least one product type. Forexample, a perishable product, e.g., a tomato, may have a differentthreshold than a non-perishable product, e.g., a lightbulb. In otherembodiments, the predefined time duration threshold is determined perretail store. For example, a decrease in product quantity of 30% mayindicate a shortage for one retail store, and a decrease in quantity of40% may indicate a shortage in another store. The identification of ashortage is discussed in further detail with reference to FIGS. 14A-14C.

The threshold product quantities and threshold time duration may bebased on one or more factors including: the type of product (e.g.,brand, category, particular product, etc.); the location of shelf withinthe store; the type of shelf (e.g., an end display, a display at aregister, or a shelf within the category area); the time of day, thenumber of employees, the size of the retail store, adjunct products, thenumber of facings of the product type, and so forth.

At step 1606, the processing device may identify at least one commonfactor contributing to product shortage duration of at least part of theplurality of occurrences of the product shortages of the at least oneproduct type in the plurality of retail stores. Common factorsassociated with product shortages and shortage durations may includeinformation related to: a schedule of arrivals of the additionalproducts (e.g., dates, times, updates from a supplier, etc.), knownorders of the additional products (e.g., quantity of scheduledproducts), delivery records (e.g., the reports on the delivery progressof the additional products), cost data (e.g., the different costsassociated with the additional products), calendar data (e.g., holidays,sport games, etc.), historical product turnover data (e.g., the amountof product sold in a particular time period), average turnover (e.g.,per hour in a day, per day in a week, per special events), employee data(e.g., staffing levels, shift durations, assigned tasks), and more. Insome embodiments, conditional distribution of a product shortageduration may be computed with respect to different factors. Further, insome examples, the computed conditional distribution may be analyzed toidentify the factors that contribute to longer (or shorter) productshortage durations, to product shortage durations that are above aselected threshold, and so forth. In some embodiments, correlationsbetween product shortage durations and different factors may becomputed. Further, in some examples, the computed correlations betweenthe product shortage durations and the different factors may be analyzedto identify the factors that contribute to longer (or shorter) productshortage durations.

Consistent with the present disclosure, the processing device mayperform big data analytics on the information about the productshortages. Typically, the big data analytics are executed in anautomatic or autonomous manner without using any user input during thelearning analysis. In one embodiment, the processing device may performlarge-scale data analysis on the received information image data andscheduling data to discover, detect, or learn new data. For example, thelarge-scale data analysis involves the process of examining largequantities of data (e.g., data mining, data discovery, etc.) to extractnew or previously unknown interesting data or patterns such as unusualdemand events that correlate to product shortages. The processing devicemay also perform large-scale data analysis on the stored data (e.g.,machine learning analysis, data modeling, pattern recognition,predictive analysis, correlation analysis, etc.) to predict, calculate,or identify implicit relationships or inferences within the stored data.For example, the processing device may identify at least one patterncorrelating a combination of different factors to product shortages.

At step 1608, the processing device may determine an action, associatedwith the at least one common factor, for potentially reducing productshortage durations of future shortages of the at least one product typein the plurality of retail stores. For example, based on the identifiedcommon factors and/or patterns associated with product shortages, asuggested action may be a staffing change, restocking rate change,product placement change, etc. such that the action prevents futureshortages. For example, if, at step 1606, a common factor associatedwith a shortage is the location of a product within the retail store(e.g., near a cash register), at step 1608, the determined action may beto increase the restocking rate for that product and/or location. Insome embodiments, an action may be determined by accessing a datastructure that holds a list of suggested actions for different factorsthat contribute to the product shortage duration. For example, when thecommon factor is a low number of store associates, the suggested actionmay include increasing the number of store associates. In anotherexample, when the common factor is an area in the retail stores, thesuggested action may include increasing the rate of product restockingfor that area of the retail stores. In yet another example, when thecommon factor is shelf area smaller than a selected threshold, thesuggested action may include increase the shelf area for a selectedproduct type at selected retail stores. In some embodiments, anoptimization model may be constructed based on estimated cost of actionsand estimated impact of actions on future product shortage durations,and the constructed optimization model may be used to find a set ofactions that reduce the estimated future product shortage durationswhile keeping the cost within a selected budget, for example using aconstraint optimization algorithm. In some embodiments, an action may bedetermined using suggestion determination module 1810, for example, asdescribed below.

At step 1610, the processing device may provide information associatedwith the identified action to an entity. The entity may be associatedwith a retail store, for example, an employee (human or robot) of retailstore 105, a user of system 100 that monitors retail store 105 (e.g.,market research entity 110), a processing device that control operationsassociated with retail store 105 (e.g., ERP system, WMS system, and SCMsystem), and any other device or individual capable of implementing thesuggested action in retail store 105. In some embodiments, the entitymay be an employee (human or robot) of supplier 110.

FIGS. 17A-17C illustrate example outputs based on data automaticallyderived from machine processing and analysis of images captured inretail store 105 according to disclosed embodiments. FIG. 17Aillustrates an optional output for an employee or manager of retailstore 105. FIG. 17B illustrates an optional output for a market researchentity 110. FIG. 17C illustrates an optional output for supplier 115.

FIG. 17A illustrates an example GUI for a communication device, e.g.,output devices 145C, which may be used by managers and/or employees ofretail store 105. In some embodiments, a manager may be associated witha plurality of retail stores for which product shortage information isaggregated and analyzed. The communication device may receive, from theprocessing device, the identified action determined at step 1608 ofmethod 1600. In some embodiments, the communication interface isconfigured to provide the information associated with the identifiedaction to a communication device associated with a managing entity ofthe plurality of retail stores. The provided information may include alikelihood that the determined action will reduce product shortagedurations of future shortages. The likelihood may be determined, forexample, based on big data analysis of the image data received from theplurality of retail stores.

In some embodiments, the information provided to the communicationdevice may include a prediction that a product shortage of a certainproduct type is about to occur. For example, the prediction may bedetermined based on an application of one or more big data analysistechniques to image data, employee data, supply chain data received fromsupplier 110 and/or stored in database 1514, marketing data receivedfrom market research entity 110, and the like. For example, dataprocessing module 1508 may use various data analysis techniques,described above, to determine that, based on historic data, before acertain holiday some products tend to be sold out. In response to theprediction, the communication device may display an alert 1704 to theretail store manager. The alert may be a pop-up window or may be sentvia one or more notification systems, e.g., to a handheld user device.

For example, as shown in FIG. 17A, action suggestion module 1510 maydetermine a number of actions to prevent a shortage or reduce theduration of an ongoing shortage. These suggested actions may bedisplayed to the user of the communication device (e.g., action 1702).In some embodiments, the processing device may predict an outcome if noaction is taken by management of the retail store. Action suggestionmodule 1510 may also determine, e.g., by one or more statisticalanalysis methods, a likelihood associated with the identified action.For example, the determined likelihood may indicate a percent chancethat the identified action will reduce the frequency and/or duration offuture shortages. For example, the processing device may determine, andGUI 1700 may indicate, that there is 85% change that increasing thenumber of employees for a particular shift will reduce the durationfuture shortages. In another example, the processing device maydetermine that there is 50% chance that changing the stocking times willreduce future shortages. Thus, a retail manager may use the likelihoodto determine the benefit of taking a particular action.

In another embodiment, data analysis module 1508 may predict a level ofeffectiveness for the determined action in reducing product shortagedurations of future shortages of the at least one product type. Theprovided information, e.g., the information displayed by thecommunication device, may include an identification of the determinedaction and its level of effectiveness. For example, a GUI may display anindication that adding one employee to a night shift may decrease theamount of product shortage will by 25%.

FIG. 17B illustrates an example graphical user interface (GUI) 1706 fora communications device (e.g., output device 145A), representative of aGUI that may be used by market research entity 110. Consistent with thepresent disclosure, market research entity 110 may assist supplier 115and other stakeholders in identifying trends, optimizing product sales,and/or implementing marketing strategies across a large number of retailstores 105.

GUI 1706 may provide market research entity 110 with information aboutproduct shortages, common factors of product shortages, and/or shortagetrends associated with a plurality of retail stores. As discussed withreference to step 1610, the action identified by action suggestionmodule 1510 may be provided to a market research entity. For example,the communication interface (e.g., external communication module 1506)is configured to provide information associated with the identifiedaction to a communication device associated with a marketing entity. Theinformation may include a prediction how a change in a shelf sizeallocated to the at least one product type will reduce product shortagedurations of future shortages. For example, the prediction may beprovided as a chart, table, infographic, and/or graph 1708. For example,for a product in high demand, a marketing manager may determine adifferent shelf size is needed to minimize shortages and push thehigh-demand product.

In another example, data analysis module 1508 may receive geographicdata from the plurality of retail stores, including locations of retailstores in which a product shortage of at least one product type hasoccurred. Data analysis module 1508 may identify at least one commonfactor based on analysis of the image data and the geographic data. GUI1706 may display geographic data, for example, as a table 1710, showinga number of shortages per geographic region. In other embodiments,geographic data may be displayed in a chart or as an infographic. Themarket research entity 110 may use the geographic information to makedecisions about current or future marketing efforts.

In some embodiments, GUI 1706 may also display a breakdown of commonfactors (bar graph 1712). Exemplary common factors associated with oneor more shortages at the plurality of stores may be that less than tenshelves were allocated to the product, the restock rate was one restockevery four hours, and the average shelf size on which the product wasdisplayed was less than one square foot. These common factors may beused, for example, by market research entity 110 and/or supplier 115 indetermining contract requirements for retail stores to reduce shortages.

FIG. 17C illustrates an example graphical user interface (GUI) 1714 fora communications device (e.g., output device 145B), representative of aGUI that may be used by supplier 115.

GUI 1714 may provide supplier 115 with information about productshortages, common factors of product shortages, and/or shortage trendsassociated with a plurality of retail stores. GUI 1714 may providesupplier 115 with an identified action to reduce shortages as determinedusing method 1600. In some embodiments, the communication interface isconfigured to provide the information associated with the identifiedaction to a communication device associated with a supplier of at leastone product type, including an indication that the product shortage isin noncompliance with contractual agreements of a retail store. Forexample, table 1716 provides an exemplary list of non-compliant storesand lists a contact name of an employee who handles contractualagreements with supplier 115. Other information may be displayedincluding the non-compliant terms, an action to bring the store backinto compliance, an option to provide the retail store with anotification of non-compliance, etc.

In another embodiment, data analysis module 1508 may receive inventorydata from the plurality of retail stores, including details about achain-of-supply of products of at least one product type associated withtimes in which product shortages of at least one product type occur. Forexample, chain-of-supply information may include delivery times, spacein storage, etc. Data analysis module 1508 may identify at least onecommon factor based on analysis of the image data and the inventorydata.

In another example, data analysis module 1508 may access historical dataassociated with the plurality of retail stores. The historical data maybe indicative of time periods during which a product shortage of atleast one product type has occurred. Data analysis module 1508 mayidentify at least one common factor based on analysis of the image dataand the historical data. A common factor may be, for example, a nationalholiday. The action identified by action suggestion module 1510 forpotentially reducing product shortage durations of future shortages ofthe at least one product type may include changing an element in thechain-of-supply of products of the at least one product type.

As an example, GUI 1714 may display the output of data analysis module1508 and action suggestion module 1510 in graphs 1718 and 1720,respectively. Data analysis module 1508 may identify certain date rangesas being a common factor in shortages of a particular product. Based onthis information, predicted shortages may be displayed in graph 1718.GUI 1714 may also display a recommended amount of deliveries based onthe predicted shortages. For example, graph 1720 may provide arecommended delivery schedule. In some embodiments, as shown in FIG.17A, GUI 1714 may also provide a likelihood of the recommended deliveryschedule to prevent product shortages.

Managing Inventory of Perishable Products

In the retail world, managing perishable products such as vegetables,fruits, dairy products, meat products, and so forth, has specificchallenges in addition to the typical challenges of managing retailproducts. Some of these challenges result from the relatively shortshelf life and varied quality of the perishable products. Retail storesrecognize that they need to treat perishable products differently thanother type of retail products. Sophisticated retail stores may maximizeprofits from perishable products by considering the state of perishableproducts currently on display, the state of perishable products instorage areas, and/or the state of perishable products scheduled toarrive. In the context of the present disclosure, the state ofperishable products may include the quality of the products, thequantity of the products, or both. In an example embodiment for managinginventory of perishable products in retail stores, a retail store maylearn about an upcoming shipment of products from a specific type ofperishable product and reduces a price of the specific type ofperishable product to avoid an unwanted inventory of products with ashort shelf life. This embodiment is described in greater detail belowwith reference to FIGS. 18-21. In another embodiment for managinginventory of perishable products in retail stores, a retail store mayinitiate restocking shelves of perishable products in some cases andforgo restocking shelves of perishable products at other cases. Thisembodiment is described in greater detail below with reference to FIGS.22-23. In yet another embodiment for managing inventory of perishableproducts in retail stores, a retail store may initiate a clearance saleof perishable products from a first type based on their state and thestate of other perishable products from a second type. This embodimentis described in greater detail below with reference to FIGS. 24-25. Thedisclosed systems and methods may enable retail stores to maximize theirprofits from perishable products by using image data from imagecapturing devices 125 (e.g., a plurality of cameras fixedly mounted toshelves and other objects in the perishable products section) andscheduling data about additional products scheduled to be delivered tothe retail stores.

FIG. 18 illustrates an exemplary embodiment of a memory device 1800containing software modules consistent with the present disclosure. Inparticular, as shown, memory device 1800 may include a retail storecommunication module 1802, a captured data analysis module 1804, anexternal sources communication module 1806, a scheduling data analysismodule 1808, a suggestion determination module 1810, a database accessmodule 1812, and a database 1814. Modules 1802, 1804, 1806, 1808, 1810,and 1812 may contain software instructions that are executable by atleast one processor (e.g., processing device 202) associated with system100. Retail store communication module 1802, captured data analysismodule 1804, external sources communication module 1806, scheduling dataanalysis module 1808, suggestion determination module 1810, databaseaccess module 1812, and database 1814 may cooperate to perform multipleoperations. For example, retail store communication module 1802 mayreceive image data from a plurality of image capturing devices 125 inretail store 105, captured data analysis module 1804 may use thereceived image data to determine information about a displayed inventoryof a plurality of perishable products, external sources communicationmodule 1806 may obtain scheduling data about additional perishableproducts scheduled to arrive to the retail store (e.g., from supplier115), scheduling data analysis module 1808 may use the receivedscheduling data to determine information about a predicted inventory ofthe plurality of perishable product, and suggestion determination module1810 may determine at least one suggestion regarding the plurality ofperishable products in retail store 105, for example, a suggestion tochange prices.

In some embodiments, memory device 1800 may be part of system 100, forexample, memory device 226. Alternatively, memory device 1800 may bestored in an external database or an external storage communicativelycoupled with server 135, such as one or more databases or memoriesaccessible over communication network 150. Further, in otherembodiments, the components of memory device 1800 may be distributed inmore than one server and more than one memory device.

In some embodiments, retail store communication module 1802 may receiveinformation from sensors located in retail store 105. In one example,retail store communication module 1802 may receive images captured by aplurality of image sensors fixedly mounted in retail store 105 orreceive data derived from images captured by plurality of image sensorsfixedly mounted in retail store 105. In another example, retail storecommunication module 1802 may receive image data (e.g., images or dataderived from images) from robotic capturing devices configured tonavigate autonomously within retail store 105 and to capture images ofmultiple types of perishable products. In yet another example, retailstore communication module 1802 may receive data from one or more shelfsensors configured to measure properties of perishable products placedon a store shelf. The one or more shelf sensors may include pressuresensitive pads, light detectors, weight sensors, light sensors, odorsensors, resistive sensors, ultrasonic sensors, and more.

In some embodiments, captured data analysis module 1804 may process theinformation collected by retail store communication module 1802 todetermine one or more existing-products (EP) indicators for theperishable products located in retail store 105. The term“existing-products indicators” or “EP indicators” may refer to dataindicative of one or more attributes associated with perishable productsfound in retail store 105. A portion of the perishable products found inretail store 105 may be located in a display area accessible tocustomers, while another portion of the perishable products found inretail store 105 may be located in a storage area not accessible tocustomers. Examples of EP indicators generated by captured data analysismodule 1804 are illustrated in FIG. 19 and described in detail below.The term “storage area” broadly includes storage areas of products notdisplayed or otherwise for sale in the retail store and may include, forexample, high shelves (e.g., storage for pallets without displayedpricing information), below display areas of a shelving unit, in adedicated storage area, and more. In one embodiment, captured dataanalysis module 1804 may determine EP indicators for multiple types ofperishable products based on image data received from a plurality ofimage sensors fixedly mounted in retail store 105 (e.g., as illustratedin FIG. 4A). In another embodiment, captured data analysis module 1804may determine EP indicators for multiple types of perishable productsusing a combination of image data and data from one or more shelfsensors configured to measure properties of perishable products placedon a store shelf (e.g., as illustrated in FIG. 8A). For example, thecaptured data analysis module 1804 may analyze the data received fromdetection elements attached to store shelves, alone or in combinationwith images captured in retail store 105, to determine the EP indicatorsfor multiple types of perishable products. In another embodiment,captured data analysis module 1804 may determine EP indicators formultiple types of perishable products using a combination of image dataand information about perishable products available in at least onestorage area of retail store 105. For example, the captured dataanalysis module 1804 may use inventory records and image data todetermine information about the current inventory of multiple types ofperishable products. The current inventory of the multiple types mayinclude the inventory displayed for sale and the inventory stored inretail store 105.

In addition, captured data analysis module 1804 may determine thequality of perishable products by analyzing image data and other typesof captured data. For example, captured data analysis module 1804 mayapply one or more image analysis techniques to assess product ripeness,a current state of one more products, etc. For example, a machinelearning model may be trained using training examples to determine aquality of perishable products from images and/or videos and/or othertypes of captured data, and the trained machine learning model may beused to analyze the image data and/or the other captured data todetermine the quality of perishable products. An example of suchtraining example may include an image and/or data of other typestogether with an indication of the quality of the perishable productsassociated with the image and/or the data. In another example, anartificial neural network (such as a deep neural network, aconvolutional neural network, etc.) may be configured to determinequality of perishable products from images and/or videos and/or othertypes of captured data, and the artificial neural network may be used toanalyze the image data and/or the other captured data to determine thequality of perishable products. Captured data analysis module 1804 mayalso analyze textual information derived from image data, analyze avisual appearance of the perishable products, calculate a duration thata perishable product has been on a shelf, determine turnover of productsfrom the product type on the shelf, and determine storage requirementsassociated with the perishable products. In some cases, determining theEP indicators for multiple types of perishable products may includeaggregating data from two or more locations in retail store 105. Forexample, in one embodiment, captured data analysis module 1804 maymonitor the quality of a type of perishable product displayed on a firstshelving unit, on a second shelving unit, and/or in a storing area. Forexample, the first shelving unit may be located in a specific aisle, thesecond shelving unit may be located in a display next to the cashier,and the storing area may be in a cold room.

In some embodiments, external sources communication module 1806 mayobtain scheduling data about additional perishable products scheduled toarrive at the retail store. In one embodiment, external sourcescommunication module 1806 may receive the scheduling data from one ormore of entities in the supply chain. In one example, the schedulingdata may be received from online services (e.g., from a server thatstore data on shipments orders). In another example, the scheduling datamay be received from supplier 115 associated with the perishableproducts (e.g., from a farmer that produced the perishable products). Inanother example, the scheduling data may be received from a marketresearch entity 110 (e.g., statistics about demand for certainperishable products). In another example, the scheduling data may bereceived from a shipment company that delivers the perishable products(e.g., from an Internet-of-Things (IoT) sensor in a cargo ship). Inanother example, the scheduling data may be received from a distributioncompany that delivers the perishable products (e.g., from an agent whosupplies the perishable products to retail stores). In one embodiment,external sources communication module 1806 may determine the schedulingdata about the additional perishable products using informationcollected from one or more of entities in the supply chain and otherdata sources, for example, Enterprise Resource Planning (ERP), WarehouseManagement Software (WMS), and Supply Chain Management (SCM)applications. In addition, external sources communication module 1806may generate the scheduling data using analytics of data associated withpast delivery of perishable products. For example, the external sourcescommunication module 1806 may determine that a shipment of avocados isscheduled to be delivered to retail store 105 based on past deliveries.

In some embodiments, scheduling data analysis module 1808 may processthe information collected by external sources communication module 1806to determine one or more scheduled-products (SP) indicators forperishable products not currently located in retail store 105. The term“scheduled products indicators” or “SP indicators” refers to dataindicative of one or more attributes associated with perishable productsscheduled to be delivered to retail store 105. Examples of SP indicatorsgenerated by scheduling data analysis module 1808 are illustrated inFIG. 19 and described in detail below. In one embodiment, schedulingdata analysis module 1808 may determine SP indicators for multiple typesof perishable products based on the scheduling data received from aplurality of entities in the supply chain. Consistent with the presentdisclosure, scheduling data analysis module 1808 may use artificialneural networks, convolutional neural networks, machine learning models,image regression models, and other processing techniques to determinethe SP indicators. For example, scheduling data analysis module 1808 mayuse a regression model on the received scheduling data to predict thestate of the perishable products when they are scheduled to be deliveredto retail store 105.

In some embodiments, suggestion determination module 1810 may use theoutput of captured data analysis module 1804 and scheduling dataanalysis module 1808 to determine at least one suggestion regardingplacement of perishable products in the retail store. The term“suggestion regarding the placement of perishable products” may includea recommendation associated with the manner in which the perishableproducts are to be located in the retail store. It includes the locationof the perishable products, the price of the perishable products, therestocking rate of the perishable products, the promotion assigned tothe perishable products, and more. Generally, the purpose of thesuggestion is to maximize profit from the perishable products. Examplesof suggestions generated by suggestion determination module 1810 areillustrated in FIG. 19 and described in detail below. In someembodiments, the suggestion determination module 1810 may determine atleast one EP suggestion (i.e., a suggestion about the placement ofexisting products) and at least one SP suggestion (i.e., a suggestionabout the placement of scheduled products). The suggestion determinationmodule 1810 may use retail store communication module 1802, externalsources communication module 1806, or both to provide the at least onedetermined suggestion to an entity associated with retail store 105. Inone example, suggestion determination module 1810 may determine that achange in the price of an existing product type is preferred and providea recommended price to a processing device associated with retail store105. In a second example, suggestion determination module 1810 maydetermine that a revision of an order from a specific supplier isrecommended and provide the revised order to a processing deviceassociated with the specific supplier.

In some embodiments, database access module 1812 may cooperate withdatabase 1814 to retrieve stored scheduling data such as product supplyinformation. The product supply information may include, for example, aschedule of arrivals of additional products, inventory records, checkoutdata, calendar data, and/or historical product turnover data. Captureddata analysis module 1804 and scheduling data analysis module 1808 mayuse the product supply information stored in database 1814 to determineone or more indicators. Database 1814 may include separate databases,including, for example, a vector database, raster database, tiledatabase, viewport database, and/or a user input database, configured tostore data. The data stored in database 1814 may be received frommodules 1802-1812, server 135, from any communication device associatedwith retail stores 105, market research entity 110, suppliers 115, andusers 120. Moreover, the data stored in database 1814 may be provided asinput using data entry, data transfer, or data uploading.

Modules 1802-1812 may be implemented in software, hardware, firmware, amix of any of those, or the like. For example, if the modules areimplemented in software, the modules may be stored in a server (e.g.,server 135) or distributed over a plurality of servers. In someembodiments, any one or more of modules 1802-1812 and data associatedwith database 1814 may be stored in database 140 and/or located onserver 135, which may include one or more processing devices. Processingdevices of server 135 may be configured to execute the instructions ofmodules 1802-1812. In some embodiments, aspects of modules 1802-1812 mayinclude software, hardware, or firmware instructions (or a combinationthereof) executable by one or more processors, alone, or in variouscombinations with each other. For example, modules 1802-1812 may beconfigured to interact with each other and/or other modules of server135 to perform functions consistent with disclosed embodiments.

Reference is now made to FIG. 19, which illustrates example outputs ofat least one processing device of system 100 (e.g., processing device202) associated with captured data analysis module 1804, scheduling dataanalysis module 1808, and suggestion determination module 1810. As shownin the figure, captured data analysis module 1804 may output one or moreEP indicators (i.e., data indicative of attributes associated withperishable products currently existing in retail store 105); schedulingdata analysis module 1808 may output one or more SP indicators (i.e.,data indicative of attributes associated with perishable productsscheduled to be delivered to retail store 105); and suggestiondetermination module 1810 may output at least one or more suggestions.Consistent with the present disclosure, suggestion determination module1810 may use at least one EP indicator and at least one SP indicator togenerate at least one suggestion.

The at least one processing device of system 100 may use captured dataanalysis module 1804 to output an EP quality indicator 1900, an EPquantity indicator 1902, an EP demand indicator 1904, an EP priceindicator 1906, and an EP planogram indicator 1908. The EP indicatorsmay be determined based on images of the at least one shelving unit(e.g., from image capturing devices 125 of FIGS. 4A-4C explained above),from data from shelf sensors such as the detection elements illustratedin FIGS. 8A and 8B, or from data aggregated from multiple sources.Additionally, the EP indicators may reflect attributes of perishableproducts currently in a display area of retail store 105, in a storagearea of retail store 105, or both. In some retail stores 105, perishableproducts may be organized in one or more piles on a shelf. Unlike manyother products, perishable products of the same type may differ from oneanother, for example, in weight, volume, shape, quality, condition,colors, and so forth. As a result, determining exact EP indicators fromimages of perishable products may be difficult. For example, a firstpile of perishable products of a certain type may include perishableproducts with a first specific size, while a second pile of perishableproducts of the certain type may include perishable products of thecertain type with a second specific size. Therefore, the at least oneprocessing device of system 100 may additionally determine and assign acertainty level for each of the determined EP indicators.

In some embodiments, the at least one processing device may use captureddata analysis module 1804 to determine one or more EP quality indicators1900 for multiple types of perishable products. Specifically, the atleast one processing device may determine EP quality indicators 1900 forthe multiple types of perishable products based on analysis of datacaptured in retail store 105. Thereafter, the at least one processingdevice may use a determined EP quality indicator 1900 and at least oneSP indicator to determine at least one suggestion regarding placement ofperishable products in the retail store. For example, when EP qualityindicator 1900 indicates that the current inventory of peaches is of lowquality and the at least one processing device knows that anothershipment of peaches is scheduled within a predetermined time period(e.g., within the next 8 hours), the at least one processing device mayrecommend clearing the peaches from the display area.

Consistent with the present disclosure, the at least one processingdevice may use Artificial Intelligence (AI) and Machine Learning (ML)algorithms to identify a condition of at least part of the perishableproducts and to determine EP quality indicator 1900 from the identifiedcondition. For example, the at least one processing device may determinethat a condition of some perishable products is rotten. For example, amachine learning model may be trained using training examples toidentify a condition of perishable products from images and/or videos,and the trained machine learning model may be used to analyze an imageof the perishable products to identify the condition of the perishableproducts. An example of such a training example may include an image ofperishable products together with an indication of the condition of theperishable products depicted in the image. In another example, anartificial neural network (such as a deep neural network, aconvolutional neural network, etc.) may be configured to identify acondition of perishable products from images and/or videos, and theartificial neural network may be used to analyze an image of theperishable products to identify the condition of the perishableproducts. In one embodiment, EP quality indicator 1900 may be associatedwith a duration that the perishable products stayed on the shelf. Insome cases, a time threshold may be selected, for example, based on atype of the perishable product, the condition of the perishable product,etc. In these cases, EP quality indicator 1900 may be determined basedon the period of time that perishable products stayed on the shelf(e.g., for a specific product type between 0 and 8 hours—high quality,between 8 hours and 24 hours—medium quality, between 24 hours and 36hours—low quality). In some cases, statistics about the duration theperishable products stayed on the shelf may be generated and provided toan employee of retail store 105 or to an online customer. In anotherembodiment, EP quality indicator 1900 may be associated with theturnover of perishable products from different parts of the shelf. Forexample, the turnover may be calculated for different regions of theshelf (such as the front region facing the customers, middle region,back region, left region, right region, grid of regions, etc.), fordifferent heights in the pile, and so forth. For example, turnover ofthe perishable products may be calculated by identifying the regions thecustomers pick perishable products from, aggregating the number ofproducts picked from each region. In another example, changes in thestructure of the pile may be identified used to calculate the turnoverfor the different regions. In another embodiment, EP quality indicator1900 may be associated with a quantity associated with customersreturning perishable products to the pile (i.e., returning a perishableproduct after picking it from the pile). For example, the ratio ofproducts returned to the pile of the products picked relative toproducts not returned to the pile may be calculated, for example, as afunction of a type of product, location of shelf, time in day, conditionof the perishable products, and so forth. The number of productsreturned to the pile may be calculated for selected time duration andmay be indicative to the quality of the perishable products. Consistentwith the present disclosure, a notification may be provided when thenumber and/or the ratio of products returned to the pile exceed aselected threshold. For example, the notification may include asuggestion to remove some of the perishable products that are of poorcondition from the pile. In another embodiment, EP quality indicator1900 may be associated with a quality estimation determined using onhyperspectral imaging technology. Specifically, image capturing device125 (e.g., a robot that roams in certain sections of the retail store)may be equipped with one or more dedicated image sensors that canmeasure light reflectance or transmittance at individual wavelengths ornarrow wavebands for determining the chemical constituents of theperishable products. The chemical constituents of the perishableproducts may be used to determine at least one trait (e.g., ripeness)associated with EP quality indicator 1900 of the perishable products. Inone example, image capturing devices 125 may capture visible andnear-infrared spectroscopy, which covers the spectral region between 400nm and 2500 nm, for chemical analysis and quality assessment ofdifferent types of perishable products in retail store 105.

In some embodiments, the at least one processing device may use captureddata analysis module 1804 to determine one or more EP quantityindicators 1902 for multiple types of perishable products. For example,the at least one processing device may determine EP quantity indicators1902 for the multiple types of perishable products based on analysis ofdata captured in retail store 105, based on analysis of data capturedfrom other sensors (such as weight sensors, pressure sensors, lightsensors, etc.) as described above, and so forth. Thereafter, the atleast one processing device may use a determined EP quantity indicator1902 and at least one SP indicator to determine at least one suggestionregarding placement of perishable products in retail store 105. Forexample, when EP quantity indicator 1902 indicates that there are alarge number of peaches currently in the store (in the display area andin the storage area) and the at least one processing device knows thatanother shipment of peaches is scheduled tomorrow, the at least oneprocessing device may recommend lowering the price of the peachescurrently in the store.

Consistent with the present disclosure, the at least one processingdevice may use AI and ML algorithms to estimate the quantity of theperishable products on display and/or in the storage area and todetermine EP quantity indicator 1902. For example, a machine learningmodel may be trained using training examples to identify a quantity ofperishable products from images and/or videos, and the trained machinelearning model may be used to analyze an image of the perishableproducts to identify the quantity of the perishable products. An exampleof such a training example may include an image of perishable productstogether with an indication of the quantity of the perishable productsdepicted in the image. In another example, an artificial neural network(such as a deep neural network, a convolutional neural network, etc.)may be configured to identify a quantity of perishable products fromimages and/or videos, and the artificial neural network may be used toanalyze an image of the perishable products to identify the quantity ofthe perishable products. In one example, the estimated quantity mayinclude a number that corresponds to the number of products from the atleast one product type. In another example, the estimated quantity maybe an indicator of the existing quantity of the at least one producttype relative to the desired quantity of the at least one product typeas identified in a relevant planogram (e.g., 65% of planogram inventoryof tomatoes). In one embodiment, EP quantity indicator 1902 may beassociated with a volume of the perishable products. Specifically, theat least one processing device may analyze images to determine arepresentative volume of a product of the perishable products, and usethe determined representative volume to estimate the total volume of theperishable products in a pile, for example, by multiplying an estimatednumber of products by the estimated volume. In another example, the atleast one processing device may analyze captured images to detect alabel associated with the perishable products, the depiction of thelabel in the one or more images may be used to determine arepresentative volume of a product of the perishable products, and therepresentative volume may be used to estimate the total volume of theperishable products in the pile. In yet another example, therepresentative volume of a product of the perishable products may beobtained from a user associated with retail store 105. In anotherembodiment, EP quantity indicator 1902 may be associated with a weightof the perishable products. For example, the at least one processingdevice may analyze images to determine a representative weight of aproduct of the perishable products and use the representative weight toestimate the total weight of the perishable products in a pile, forexample, by estimating the number of perishable products in the pile andmultiplying it by the estimated weight. In yet another example, arepresentative weight of a product of the perishable products may beobtained from a user associated with retail store 105, and the obtainedweight may be used to estimate the total weight of the perishableproducts in the pile. In addition, the at least one processing devicemay use weight measurements from additional sensors (e.g., detectionelement 851) to determine the total weight of the perishable products inthe pile and use this information to determine EP quantity indicator1902. Alternatively, the estimated total volume of the perishableproducts in a pile may be obtained (for example, as described above),and the representative weight may be used together with the total volumeto estimate the quantity of the perishable products in retail store 105.

In some embodiments, the at least one processing device may use captureddata analysis module 1804 to determine one or more EP demand indicators1904 for multiple types of perishable products. Specifically, the atleast one processing device may determine EP demand indicators 1904 forthe multiple types of perishable products based on analysis of datacaptured in retail store 105. Thereafter, the at least one processingdevice may use EP demand indicator 1904 and at least one SP indicator todetermine at least one suggestion regarding placement of perishableproducts in retail store 105. For example, when EP demand indicator 1904indicates that there is a high demand for peaches and another shipmentof peaches is scheduled for the day after tomorrow, the at least oneprocessing device may recommend increasing the price of the peachescurrently in the store. Consistent with the present disclosure, the atleast one processing device may use AI and ML algorithms to estimate thedemand for different types of perishable products and to determine EPdemand indicator 1904. In one embodiment, EP demand indicator 1904 maybe associated with a number of customers standing next to at least oneshelving unit associated with a specific type of perishable productswithin a selected time duration. For example, the at least oneprocessing device may analyze captured images to determine the number ofcustomers standing next to a shelving unit. For example, a machinelearning model may be trained using training examples to determine thenumber of people in a selected region of the environment from imagesand/or videos, and the trained machine learning model may be used toanalyze the captured images and determine the number of customersstanding next to a shelving unit. An example of such a training examplemay include an image with an indication of a selected region in theimage, together with an indication of the number of people within theselected region. In another example, an artificial neural network (suchas a deep neural network, a convolutional neural network, etc.) may beconfigured to determine the number of people in a selected region of theenvironment from images and/or videos, and the artificial neural networkmay be used to analyze the captured images and determine the number ofcustomers standing next to a shelving unit. In another embodiment, EPdemand indicator 1904 may be associated with the number of customerspicking up perishable products from a shelving unit associated with thespecific type of perishable products during a selected time duration.For example, the at least one processing device may analyze the capturedimages to determine the number of customers picking up perishableproducts from the shelving unit. For example, a machine learning modelmay be trained using training examples to determine the number ofcustomers picking up perishable products from selected shelving unitsusing images and/or videos, and the trained machine learning model maybe used to analyze the captured images and determine the number ofcustomers picking up perishable products from the shelving unit. Anexample of such a training example may include an image with anindication of a selected shelving unit, together with an indication ofthe number of customers picking up perishable products from the shelvingunit. In another example, an artificial neural network (such as a deepneural network, a convolutional neural network, etc.) may be configuredto determine the number of customers picking up perishable products fromshelving units using images and/or videos, and the artificial neuralnetwork may be used to analyze the captured images and determine thenumber of customers picking up perishable products from the shelvingunit. In another embodiment, EP demand indicator 1904 may be associatedwith a quantity of perishable products that each costumer picks from atleast one shelving unit associated with the specific type of perishableproducts. For example, the at least one processing device may analyzethe captured images to determine the quantity of products that eachcostumer picks from at least one shelving unit. For example, a machinelearning model may be trained using training examples to determine thequantities of products that different costumer picks from selectedshelving units using images and/or videos, and the trained machinelearning model may be used to analyze the captured images and determinethe quantity of products that each costumer picks from the shelvingunit. An example of such a training example may include images with anindication of a selected shelving unit, together with an indication ofthe quantity of products that each costumer picks from the shelvingunit. In another example, an artificial neural network (such as a deepneural network, a convolutional neural network, etc.) may be configuredto determine the quantities of products that different costumer picksfrom selected shelving units using images and/or videos, and theartificial neural network may be used to analyze the captured images anddetermine the quantity of products that each costumer picks from theshelving unit. In some embodiments, the at least one processing devicemay use captured data analysis module 1804 together with data from othersources to determine EP demand indicators 1904 for multiple types ofperishable products located in retail store 105. In one example, EPdemand indicator 1904 may be associated with the number of searches thatonline customers made for a specific type of perishable products withina selected time duration. In another example, EP demand indicator 1904may be associated with sales data during a selected time duration.

In some embodiments, the at least one processing device may use captureddata analysis module 1804 to determine one or more EP price indicators1906 for one or more types of perishable products. For example, the atleast one processing device may determine price indicators 1906 for oneor more types of perishable products based on analysis of data capturedin retail store 105, based on information from a database that containprices for different products, and so forth. Thereafter, the at leastone processing device may use a determined EP price indicators 1906 andat least one SP indicator to determine at least one suggestion regardingplacement of perishable products in retail store 105. For example, theEP price indicators 1906 may include prices of one or more perishableproducts, may include indication of whether the prices of one or moreperishable products are below or above a selected threshold, may includean indication of whether one or more perishable products are on specialsale, and so forth. For example, when the price of a particularperishable product is below a selected threshold and the SP indicatorsindicates that another shipment of the particular perishable product isnot scheduled for at least a selected time duration, the at least oneprocessing device may recommend increasing the price of the particularperishable product. In some examples, images captured from the retailstore may be analyzed to determine a price of a particular perishableproduct. For example, an OCR algorithm may be used to extract textualinformation from a price label associated with the particular perishableproduct.

In some embodiments, the at least one processing device may use captureddata analysis module 1804 to determine one or more EP planogramindicators 1908 for multiple types of perishable products. For example,the at least one processing device may determine planogram indicators1908 for the multiple types of perishable products based on analysis ofdata captured in retail store 105, based on input from other sensors(such as weight sensors, pressure sensors, light sensors, etc., forexample as described above in relation to processes 1000 and 1050), andso forth. Thereafter, the at least one processing device may use adetermined EP planogram indicator 1908 and at least one SP indicator todetermine at least one suggestion regarding placement of perishableproducts in retail store 105. For example, when planogram indicator 1908indicates that a temporary size of a display area for peaches is lowerthan an original display area allocated to peaches (e.g., with respectto a planogram, with respect to a realogram, etc., e.g., due to shortageof peaches) and the at least one processing device knows that anothershipment of peaches is scheduled only the day after tomorrow, the atleast one processing device may recommend increasing the display area ofa different product. Consistent with the present disclosure, the atleast one processing device may analyze captured images to determine ameasurement of planogram compliance associated with a specific type ofperishable products (e.g., with respect to a planogram, with respect toa realogram, etc.) and determine the planogram indicators 1908 based onthe measurement of planogram compliance.

As further illustrated in FIG. 19, the at least one processing device ofsystem 100 may use scheduling data analysis module 1808 to output an SPquality indicator 1910, an SP quantity indicator 1912, an SP demandindicator 1914, an SP cost indicator 1916, an SP shelf-life indicator1918, and an SP ETA indicator 1920. The SP indicators may be determinedbased on scheduling data received from one or more sources (e.g.,entities in the supply chain). Additionally, the SP indicators mayreflect attributes of perishable products currently not located inretail store 105, but scheduled to be delivered to retail store 105.

In some embodiments, the at least one processing device may usescheduling data analysis module 1808 to determine one or more SP qualityindicators 1910 for the additional perishable products scheduled to bedisplayed on at least one shelving unit in retail store 105.Specifically, the at least one processing device may determine apredicted quality for multiple types of perishable products scheduled tobe delivered to retail store 105 based on data obtained by externalsources communication module 1806. Thereafter, the at least oneprocessing device may use at least one EP indicator and SP qualityindicator 1910 to determine at least one suggestion regarding placementof perishable products in retail store 105. For example, when SP qualityindicator 1910 indicates that the condition of the bananas in anupcoming shipment is already ripe and the at least one processing deviceknows that there is a large number of ripe bananas in retail store 105,the at least one processing device may recommend lowering the price ofthe bananas currently in retail store 105. Consistent with the presentdisclosure, the at least one processing device may use product qualityinformation associated with additional perishable products and theestimated arrival time of the additional perishable products to predictthe quality of the additional perishable products at the time they areestimated to arrive to retail store 105. The at least one processingdevice may use a regression model on the product quality information andpossibly additional inputs such as the supplier's records, calendardata, the delivery conditions (e.g., using refrigerator truck), andmore. In some examples, the at least one processing device may determineSP quality indicator 1910 using a machine learning model trained topredict the quality of the additional perishable products from theinputs described above. In other examples, the at least one processingdevice may determine SP quality indicator 1910 using an artificialneural network configured to predict the quality of the additionalperishable products from the inputs described above. In one embodiment,the at least one processing device may determine a distribution ofpredicted qualities (i.e., assigning probabilities to different level ofqualities) for the additional perishable products during the time theyare estimated to arrive to retail store 105. Some examples of suchdistributions may include normal distribution, Fisher's non-centralhypergeometric distribution, Wallenius' non-central hypergeometricdistribution, Poisson distribution, and so forth.

In some embodiments, the at least one processing device may usescheduling data analysis module 1808 to determine one or more SPquantity indicators 1912 for the additional perishable productsscheduled to be displayed on at least one shelving unit in retail store105. Specifically, the at least one processing device may determine thequantity for multiple types of perishable products scheduled to bedelivered to retail store 105 based on data obtained by external sourcescommunication module 1806. Thereafter, the at least one processingdevice may use at least one EP indicator and SP quantity indicator 1912to determine at least one suggestion regarding placement of perishableproducts in retail store 105. For example, when SP quantity indicator1912 indicates that only half of the ordered bananas are scheduled toarrive and the at least one processing device knows that there is alimited number of ripe bananas in retail store 105, the at least oneprocessing device may recommend increasing the price of the bananascurrently in retail store 105. Consistent with the present disclosure,the at least one processing device may determine the quantity of theadditional perishable products scheduled to be delivered to retail store105 based on reports from the suppliers, reports from the deliverycompany, IoT sensors that accompany the additional perishable productsduring their shipments, and other sources. In one embodiment, whenupdates and reports on the additional perishable products areunavailable, the at least one processing device may predict the quantityof the additional perishable products scheduled to be delivered toretail store 105 based on orders data and past delivery data.

In some embodiments, the at least one processing device may usescheduling data analysis module 1808 to determine one or more SP demandindicators 1914 for the additional perishable products scheduled to bedisplayed on at least one shelving unit in retail store 105.Specifically, the at least one processing device may determine thepredicted demand for multiple types of perishable products scheduled tobe delivered to retail store 105 based on data obtained by externalsources communication module 1806. Thereafter, the at least oneprocessing device may use at least one EP indicator and SP demandindicator 1914 to determine at least one suggestion regarding placementof perishable products in retail store 105. For example, when SP demandindicator 1914 indicates that the predicted demand for bananas is highand the bananas in retail store 105 are displayed at a locationassociated with a low turnover, the at least one processing device mayrecommend to reposition the bananas at a location with a high turnover.Consistent with the present disclosure, the at least one processingdevice may determine the predicted demand for multiple types ofperishable products scheduled to be delivered to retail store 105 usingforecasting algorithms, including statistical algorithms such as Fourierand multiple linear regression algorithms. The forecasting algorithmsmay use a variety of factors relating to different perishable products,and various types of demand history data (e.g., shipments data,point-of-sale data, customer order data, return data, marketing data,and more). Generally, demand history data can be broken into two types:base and non-base. Base history data includes predictable demand datathat may be repeatable. Conversely, non-base history data is that partof demand that is due to special events, such as promotions or extrememarket circumstances. As such, non-base history data may be associatedwith past demand events that are not anticipated to repeat themselvesfrom one period of time to the next. However, non-base history data maystill be stored within the history database to help the system toforecast the impact of special events upon future product demand. The atleast one processing device may use one or more forecasting algorithmspaired with various type of history data and run various simulations toincrease the confidence level in determining the SP demand indicator1914. For example, predictable seasonal patterns in demand history canbe very accurately modeled by sine and cosine waves. Thus, the at leastone processing device may use Fourier type demand planning algorithmswith two to three years of demand history to generate SP demandindicator 1914.

In some embodiments, the at least one processing device may usescheduling data analysis module 1808 to determine one or more SP costindicators 1916 for the additional perishable products scheduled to bedisplayed on at least one shelving unit in retail store 105.Specifically, the at least one processing device may estimate the costsassociated with multiple types of perishable products scheduled to bedelivered to retail store 105 based on data obtained by external sourcescommunication module 1806. Thereafter, the at least one processingdevice may use at least one EP indicator and SP cost indicator 1916 todetermine at least one suggestion regarding placement of perishableproducts in retail store 105. For example, when SP cost indicator 1916indicates that the expenses associated the shipment of the bananas arehigh and the at least one processing device knows that other availablefruits (classified as alternative to bananas) in retail store 105 are inlow quality, the at least one processing device may recommend a minimumprice for the bananas scheduled to be delivered to retail store 105.Consistent with the present disclosure, the at least one processingdevice may determine recommended prices for multiple types of perishableproducts scheduled to be delivered to retail store 105 based dataobtained by external sources communication module 1806. In one example,the at least one processing device may obtain from market researchentity 110 information about competitor's prices of a type of productscheduled to be delivered to retail store 105, information about thepredicted demand for the type of product scheduled to be delivered toretail store 105, information about recent sales of the type of productscheduled to be delivered to retail store 105, and more. The at leastone processing device may use the information from market researchentity 110 to determine SP cost indicators 1916. In another example, theat least one processing device may obtain from supplier 115 informationabout costs associated with the type of product scheduled to bedelivered to retail store 105, information about the quality of the typeof product scheduled to be delivered to retail store 105, informationabout the shelf-life scheduled to be delivered to retail store 105, andmore. The at least one processing device may use the information fromsupplier 115 to determine SP cost indicators 1916. In another example,the at least one processing device may obtain from online servicesinformation about events that may affect the demand for the type ofproduct scheduled to be delivered to retail store 105, information fromone or more IoT sensors that accompany a delivery of the perishableproducts and monitor the condition of the perishable products during thedelivery. The at least one processing device may use some or all theinformation obtained from online services to determine SP costindicators 1916.

In some embodiments, the at least one processing device may usescheduling data analysis module 1808 to determine one or more SPshelf-life indicators 1918 for the additional perishable productsscheduled to be displayed on at least one shelving unit in retail store105. Specifically, the at least one processing device may estimate theshelf-life of multiple types of perishable products scheduled to bedelivered to retail store 105 based on data obtained by external sourcescommunication module 1806. Thereafter, the at least one processingdevice may use at least one EP indicator and SP shelf-life indicator1918 to determine at least one suggestion regarding placement ofperishable products in retail store 105. For example, when SP shelf-lifeindicator 1918 indicates that the bananas scheduled to be delivered toretail store 105 will be overripe in three days and processing deviceknows that retail store 105 already has a large amount of ripe bananas,the at least one processing device may recommend a maximum price for thebananas. Consistent with the present disclosure, the at least oneprocessing device may determine the shelf-life of the additionalperishable products scheduled to be delivered to retail store 105 basedon the predicted quality of the additional perishable products at theestimated delivery time. In one embodiment, the at least one processingdevice may retrieve from a database information correlating a predictedcondition of each type of perishable product with an estimatedshelf-life value. Additionally, the at least one processing device maydetermine SP shelf-life indicator 1918 using artificial neural networkand/or a machine learning model trained to predict the estimatedshelf-life of the additional perishable products from history data andother inputs described above.

In some embodiments, the at least one processing device may usescheduling data analysis module 1808 to determine one or more SP ETAindicators 1920 for additional perishable products scheduled to bedisplayed on at least one shelving unit in retail store 105.Specifically, the at least one processing device may determine theestimated arrival time of the additional perishable products scheduledto be delivered to retail store 105 based on data obtained by externalsources communication module 1806. Thereafter, the at least oneprocessing device may use at least one EP indicator and SP ETA indicator1920 to determine at least one suggestion regarding placement ofperishable products in retail store 105. For example, when SP ETAindicator 1920 indicates that the bananas scheduled to be delivered toretail store 105 in 16 hours and the at least one processing deviceknows that retail store 105 recently ran out of bananas, it mayrecommend using the display area allocated to bananas to promote salesof a different product. Consistent with the present disclosure, the atleast one processing device may receive updates from a delivery companyto determine SP ETA indicator 1920. Additionally, the at least oneprocessing device may determine SP ETA indicator 1920 using artificialneural network and/or a machine learning model trained to predict theestimated arrival time of the additional perishable products fromhistory data and other inputs described above.

As further illustrated in FIG. 19, the at least one processing device ofsystem 100 may use suggestion determination module 1810 to output an EPlocation suggestion 1922, an EP size suggestion 1924, an EP promotionsuggestion 1925, an EP reorganization suggestion 1926, an SP locationsuggestion 1928, an SP size suggestion 1930 and an SP promotionsuggestion 1932. As mentioned above, suggestion determination module1810 may use at least one EP indicator and at least one SP indicator togenerate at least one suggestion. In one embodiment, suggestiondetermination module 1810 may use a plurality of EP indicators (e.g., atleast two, at least three, at least four) and a plurality of SPindicators (e.g., at least two, at least three, at least four, or atleast five) to generate at least one suggestion. In another embodiment,suggestion determination module 1810 may use any combination of a set ofEP indicators with any set of SP indicators to determine any type ofsuggestion. The term “a set of indicators” in this disclosure refers toone or more indicators.

In some embodiments, the at least one processing device may usesuggestion determination module 1810 to determine EP location suggestion1922 that includes an indication of a temporary location for at leastsome of the plurality of perishable products to reside prior to arrivalof the additional perishable products. Similarly, the at least oneprocessing device may use suggestion determination module 1810 todetermine SP location suggestion 1928 that includes an indication of alocation for the additional perishable products after they have beendelivered to retail store 105. In particular, the at least oneprocessing device may determine the location for at least one type ofperishable products based on the information about the displayedinventory (i.e., at least one EP indicator) and the information aboutthe additional perishable products (i.e., at least one SP indicator).For example, when the determined EP quality indicator 1900 indicatesthat about 75% of the avocados in retail store 105 are ripe, and SPquality indicator 1910 indicates that about 25% of the avocadosscheduled to be delivered today to retail store 105 are also ripe, theat least one processing device may recommend placing at least some ofthe avocados in a location associated with high customer turnover (e.g.,displays in the middle of an aisle, on a corner bay stand, on the firststand the customers see when entering the department, etc.). Consistentwith the present disclosure, the at least one processing device may useAI and ML algorithms to determine a customer turnover ranking fordifferent locations in retail store 105, and to use the customerturnover ranking in determining EP location suggestion 1922 and SPlocation suggestion 1928. The determination of the customer turnoverranking may be based on image data analysis and sale data analysis. Inaddition, the at least one processing device may use subsequentlycaptured image data to monitor compliance to the EP location suggestion1922 and/or SP location suggestion 1928, find correlation between actualin store coupling to sales, and improve its AI and ML algorithmsaccordingly. In some embodiments, a conditional distribution of productturnover rank may be computed with respect to different locations.Further, in some examples, the computed conditional distribution may beanalyzed to identify the locations that contribute to higher (or lower)product turnover, to product turnover that is above a selectedthreshold, and so forth. In some embodiments, correlations betweenproduct turnover and different locations may be computed. Further, insome examples, the computed correlations between the product turnoverand the different locations may be analyzed to identify the locationsthat contribute to higher (or lower) product turnover.

In some embodiments, the at least one processing device may usesuggestion determination module 1810 to determine EP size suggestion1924 that includes an indication of a temporary size of a display areafor at least some of the plurality of perishable products to be usedprior to arrival of the additional perishable products. Similarly, theat least one processing device may use suggestion determination module1810 to determine SP size suggestion 1930 that includes an indication ofthe size of a display area for the additional perishable products afterthey have been delivered to retail store 105. In particular, the atleast one processing device may determine the suggested display area forat least one type of perishable products based on the information aboutthe displayed inventory (i.e., at least one EP indicator) and theinformation about the additional perishable products (i.e., at least oneSP indicator). For example, when the determined EP quantity indicator1902 indicates that the amount of avocados retail store 105 is 130% thana typical amount and SP ETA indicator 1920 indicates that anothershipment of avocados is scheduled to be delivered later today to retailstore 105, the at least one processing device may recommend increasingthe display area of avocados relative to another product that has longershelf life. Consistent with the present disclosure, EP size suggestion1924 and SP size suggestion 1930 may be time dependent. Specifically,based on image data analysis and/or sale data analysis, the at least oneprocessing device may determine that a predicted demand for a certaintype of perishable product during first time period is greater than thepredicted demand during a second time period, and suggest a greaterdisplay size during the first time period. For example, the at least oneprocessing device may determine that on Mondays and Wednesdays there isgreater demand for avocados relative to the other days of the week.Thus, the at least one processing device may consider the temporalchanges in the determined demand when determining EP size suggestion1924 and SP size suggestion 1930.

In some embodiments, the at least one processing device may usesuggestion determination module 1810 to determine EP promotionsuggestion 1925 and SP promotion suggestion 1932. In this disclosure,the term “promotion” may include a change in price or an advertisement.EP promotion suggestion 1925 may include an indication of a temporaryprice for at least some of the plurality of perishable products to beapplied prior to arrival of the additional perishable products.Similarly, SP promotion suggestion 1932 may include an indication of theprice for the additional perishable products after they have beendelivered to retail store 105. In particular, the at least oneprocessing device may determine the recommended promotion for at leastone type of perishable products based on the information about thedisplayed inventory (i.e., at least one EP indicator) and theinformation about the additional perishable products (i.e., at least oneSP indicator). For example, when the determined EP demand indicator 1904indicates that the demand for avocados is low and SP ETA indicator 1920indicates that another shipment of avocados is scheduled to be deliveredlater today to retail store 105, the at least one processing device mayrecommend lowering the price and of avocados and display a “sale” sign.Consistent with the present disclosure, the at least one processingdevice may use the following list of variables to determine promotionsuggestions for a type of perishable products: quantity of existingproducts, quantity of scheduled products, quality of existing products,quality of scheduled products, predicted shelf-life for existingproducts, predicted shelf-life for scheduled products, average quantitysales per day, predicted demand, retail price, costs, costs ofdiscarding the perishable products, costs of returning the perishableproducts to the supplier, and more. The price may be dynamicallydetermined for each type of perishable product based on the financialimpact. In some cases, the at least one processing may autonomouslycontrol a price display (e.g., an electronic display) to present thedetermined price for the perishable products.

In some embodiments, the at least one processing device may usesuggestion determination module 1810 to determine EP reorganizationsuggestion 1926 that includes at least one task for reorganizingperishable products in retail store 105. Specifically, the at least oneprocessing device may determine the least one task for reorganizingperishable products in retail store 105 based on information aboutperishable products available in a storage area of retail store 105(i.e., at least one EP indicator) and the information about theadditional perishable products (i.e., at least one SP indicator). In oneembodiment, EP reorganization suggestion 1926 may include a task forreorganizing perishable products currently available in the storagearea. For example, when the determined EP quantity indicator 1902indicates that there are 200 Kg of avocados in the cold store and SP ETAindicator 1920 indicates that another shipment of 500 Kg avocados isscheduled to be delivered in the next hour, the at least one processingdevice may recommend reorganizing the space in the cold store by takingthe apples out from the cold store. In another embodiment, EPreorganization suggestion 1926 may include a task for reorganizingperishable products placed on the at least one shelving unit. Forexample, when EP demand indicator 1904 indicates that the tomatoeslocated in a display area adjacent to the avocados have low demand andSP quality indicator 1910 indicates that about 50% of the avocados inthe coming shipment are ripe, the at least one processing device mayrecommend increasing the display area of the avocado relative to thetomatoes' display area. Consistent with the present disclosure, the atleast one processor may predict a condition of the perishable productsavailable in the storage area of retail store 105 at a time when theadditional perishable products are scheduled to arrive (e.g., EP qualityindicator 1900). Thereafter, the at least one processor may use theinformation about the perishable products available in the storage area,the information about the additional perishable products and thepredicted condition to determine at least one task for reorganizingperishable products in retail store 105. For example, by the time thenew shipment of avocados will arrive, the avocados in the display areawill be overripe and, therefore, the avocados in the display area shouldbe cleared.

Reference is now made to FIG. 20, which illustrates an example ofgenerating a suggestion relating to perishable products based on imagedata and scheduling data. In the illustrated example, system 100receives and analyses image data 2000 to determine at least one EPindicator. Image data 2000 may include an image depicting the displayarea of the onions and, as shown, the amount of onions left on the storeshelf is low. Thereafter, system 100 receives scheduling data 2002 andanalyzes scheduling data 2002 to determine at least one SP indicator.Scheduling data 2002 may indicate that a shipment of onions is en route.Using image data 2000 and scheduling data 2002, system 100 may determineat least one suggestion regarding placement of perishable products inretail store 105. In the following example, system 100 may cause achange in the displayed price of the onions and added a promotion sign.In one embodiment, system 100 may cause the change in the displayedprice of the onions by controlling an electronic price display. Inanother embodiment, system 100 may assign a task to a store employeeassociated with the determined suggestion.

FIG. 21 depicts an exemplary method 2100 for identifying perishableproducts and automatically generating suggestions relating to theidentified products based on analysis of image data 2000. For purposesof illustration, in the following description, reference is made tocertain components of system 100. It will be appreciated, however, thatother implementations are possible and that other components may beutilized to implement the exemplary method. It will also be readilyappreciated that the illustrated method can be altered to modify theorder of steps, delete steps, or further include additional steps.

At step 2102, a processing device (e.g., processing device 202) mayreceive a set of images depicting a plurality of perishable productsdisplayed on a shelving unit in retail store 105. In one embodiment, theset of images may be acquired by any one of image capturing devices 125illustrated in FIG. 4A-4C. For example, the set of images may beacquired by a plurality of image sensors fixedly mounted in retail store105. Details of the image capturing devices mounted in retail store 105and configured to acquire the set of images are described withreferences to FIGS. 5A-5C. The set of images may be used to derive imagedata such as processed images (e.g., cropped images, video streams, andmore) or information derived from the captured images (e.g., data aboutproducts identified in captured images, data that may be used toconstruct a 3D image, and more). In one embodiment, the processingdevice may periodically receive a set of images that corresponds withthe current inventory of the perishable products on the shelf. In thisdisclosure, the term “set of images” refers to one or more images. Forexample, the processing device may receive a set of images atpredetermined time intervals (e.g., every minute, every 5 minutes, every15 minutes, every 30 minutes, etc.). In another embodiment, theprocessing device may receive the set of images in response to adetection of a certain identified event. In one example, the processingdevice may receive the set of images after an event, such as a detectedlifting of a perishable product from the shelving unit.

At step 2104, the processing device may analyze the set of images todetermine information about a displayed inventory of the plurality ofperishable products. Consistent with the present disclosure, theprocessing device may analyze image data 2000 to determine one or moreEP indicators associated with one or more types of perishable productsin retail store 105. The one or more types may include, for example,more than 25 types of perishable products, more than 50 types ofperishable products, more than 100 types of perishable products, etc. Inone embodiment, the processing device may determine the informationabout the displayed inventory of the plurality of perishable productsusing a combination of image data derived from the received set ofimages and/or data from one or more additional sensors configured todetect properties of perishable products placed on a store shelf. Forexample, the one or more additional sensors may include weight sensors,pressure sensors, touch sensors, light sensors, odors sensors, anddetection elements as described in relation to FIGS. 8A, 8B and 9, andso forth. Such sensors may be used alone or in combination with imagecapturing devices 125. For example, the processing device may analyzethe data received from odors sensors alone or in combination with imagescaptured from the retail store, to estimate the condition or theripeness level of perishable products placed on a store shelving unit.

In some embodiments, analyzing the set of images to determineinformation about a displayed inventory of the plurality of perishableproducts may further comprise preprocessing the image data. In someexamples, the image data may be preprocessed by transforming the imagedata using a transformation function to obtain a transformed image data,and the preprocessed image data may include the transformed image data.For example, the transformation function may comprise convolutions,visual filters (such as low-pass filters, high-pass filters, band-passfilters, all-pass filters, etc.), nonlinear functions, and so forth. Insome examples, the image data may be preprocessed by smoothing the imagedata, for example, by using Gaussian convolution, using a median filter,and so forth. In some examples, the image data may be preprocessed toobtain a different representation of the image data. For example, thepreprocessed image data may include: a representation of at least partof the image data in a frequency domain; a Discrete Fourier Transform ofat least part of the image data; a Discrete Wavelet Transform of atleast part of the image data; a time/frequency representation of atleast part of the image data; a representation of at least part of theimage data in a lower dimension; a lossy compression representation ofat least part of the image data; a lossless compression representationof at least part of the image data; a time order series of any of theabove; any combination of the above; and so forth. In some examples, theimage data may be preprocessed to extract edges, and the preprocessedimage data may include information based on and/or related to theextracted edges. In some examples, the image data may be preprocessed toextract visual features from the image data. Some examples of suchvisual features include edges, corners, blobs, ridges, Scale InvariantFeature Transform (SIFT) features, temporal features, and so forth. Oneof ordinary skill in the art will recognize that the image data may bepreprocessed using other kinds of preprocessing methods.

In some embodiments, a machine learning model may be trained usingtraining examples to determine information about displayed inventoriesof perishable products from images and/or videos, and the trainedmachine learning model may be used to analyze the set of images and theinformation about a displayed inventory of the plurality of perishableproducts. An example of such a training example may include one or moreimages together with the desired information to be determined for thatone or more images. In some embodiments, an artificial neural network(such as a deep neural network, a convolutional neural network, etc.)may be configured to determine information about displayed inventoriesof perishable products from images and/or videos, and the artificialneural network may be used to analyze the set of images and theinformation about a displayed inventory of the plurality of perishableproducts.

At step 2106, the processing device may obtain information aboutadditional perishable products scheduled to be displayed on the at leastone shelving unit. Consistent with the present disclosure, theprocessing device may analyze scheduling data 2002 to determine one ormore SP indicators associated with one or more types of perishableproducts scheduled to be delivered to retail store 105. The informationabout additional perishable products may include at least one of: aschedule of arrivals of the additional products (e.g., dates, times,updates from a supplier, etc.), known orders of the additional products(e.g., quantity of scheduled products), delivery records (e.g., thereports on the delivery progress of the additional products), cost data(e.g., the different costs associated with the additional products),calendar data (e.g., holidays, sport games, etc.), historical productturnover data (e.g., the amount of avocados sold per last April),average turnover (e.g., per hour in a day, per day in a week, perspecial events), and more. In one embodiment, the information about theadditional perishable products may be obtained from multiple sources(e.g., market research entity 110, supplier 115, and multiple retailstores 105) and may include different types of parameters. In someembodiments, the obtained information about the additional perishableproducts may be stored digitally in memory device 226, stored indatabase 140, received using network interface 206 through communicationnetwork 150, received from a user, determined by processing device 202,and so forth.

At step 2108, the processing device may use the information about thedisplayed inventory and the information about the additional perishableproducts to determine at least one suggestion regarding the placement ofperishable products in retail store 105. Consistent with the presentdisclosure, the processing device may perform big data analytics on theinformation about the displayed inventory and/or the information aboutthe additional perishable products. Typically, the big data analyticsmay be executed in an automatic or autonomous manner without using anyuser input during the learning analysis. In one embodiment, theprocessing device may perform large-scale data analysis on the receivedinformation image data and scheduling data to discover, detect, or learnnew data. For example, the large-scale data analysis involves theprocess of examining large quantities of data (e.g., data mining, datadiscovery, etc.) to extract new or previously unknown interesting dataor patterns such as unusual demand events for a certain type ofperishable product. The processing device may also perform large-scaledata analysis on the stored data (e.g., machine learning analysis, datamodeling, pattern recognition, predictive analysis, correlationanalysis, etc.) to predict, calculate, or identify implicitrelationships or inferences within the stored data. For example, theprocessing device may identify at least one pattern correlating acombination of different parameters to product sales. The differentparameters may include the quantity of perishable products on display,the quality of perishable products on display, the predicted demand forthe perishable products, the location that the perishable products arebeing displayed, the time of day, the day of the week, the price ofperishable products. Later on, the identified at least one pattern maybe used for determining suggestions regarding the placement of differenttypes of perishable products in retail store 105.

At step 2110, the processing device may provide the suggestion to anentity associated with the retail store. Consistent with the presentdisclosure, the entity associated with retail store 105 may be anemployee (human or robot) of retail store 105, a costumer of retailstore 105, a user of system 100 that monitors retail store 105 (e.g.,market research entity 110), a processing device that control operationsassociated with retail store 105 (e.g., ERP system, WMS system, and SCMsystem), and any other device or individual capable of implementingsuggestions regarding placement of perishable products in retail store105.

Consistent with an embodiment of the present disclosure, system 100 maysupport different types of retail stores. For example, a first type ofretail store may have a policy that fruits, vegetables, and baked goodsstands should be full in order to provide a positive visual experience.Therefore, if a display area dedicated to one type of perishable productis below a visual threshold (e.g., the height of the pile may be too lowor the total empty space on a part of the shelf is greater than 50%) thetype of perishable product should be restocked. However, if there are nomore products from that type of perishable products in the storeinventory, the display area dedicated to that type of perishableproducts needs to be reduced in favor of another type of perishableproduct that is available. On the other hand, a second type of retailstore may have a policy to get to the end-of-day (or the end-of-week)with empty perishable products shelves to reduce the number ofperishable products with low shelf life being thrown away. For example,when system 100 predicts that a large amount of tomatoes will remain onshelf at the end-of-day, a promotion may be launched to reduce the priceof the tomatoes. In another example, when system 100 predicts that acertain type of baked product may remain on shelf at the end-of-day, asuggestion may be provided to stop baking additional products from thistype.

FIG. 22 illustrates a graph 2200 that shows the changes in quality andquantity of perishable products from a certain type in retail store 105during a workday from 6:00 AM to 10:00 PM. Graph 2200 includes threethresholds: a quality threshold at 75%, a quantity threshold at 50%, anda time threshold at 6:00 PM. As shown, system 100 triggered two productrestocking events, the first restocking event took place at t1 and thesecond restocking event took place at t2. In addition, the depictedworkday is divided to five time periods.

The first time period is from 6:00 to t1. In the first time period, thequality of the perishable products remains substantially the same, butthe quantity of the perishable products drops down and reaches thequantity threshold at t1. When system 100 determines that the quantityof the perishable products is about to reach the quantity threshold, ittriggers a first restocking event. Consistent with the presentdisclosure, system 100 determines that the quantity of the perishableproducts is about to reach the quantity threshold using the determinedEP indicators, for example, by analyzing images captured by imagecapturing devices 125 and optionally analyzing data from one or moretypes of shelf sensors.

The second time period is from t1 to t2. In the second time period, thequality of the perishable products drops down and reaches the qualitythreshold at t2, and the quantity of the perishable products drops downbut does not reach the quantity threshold. When system 100 determinesthat the quality of the perishable products is about to reach thequality threshold, it triggers a second restocking event. Consistentwith the present disclosure, system 100 determines that the quality ofthe perishable products is about to reach the quality threshold usingthe determined EP indicators, for example, by analyzing images capturedby image capturing devices 125 and optionally analyzing data from one ormore types of shelf sensors.

The third time period is from t2 to t3. In the third time period, thequality of the perishable products drops down but does not reach thequality thresholds and the quantity of the perishable products dropsdown and reaches the quantity threshold at t3. In addition, during thethird time period the time threshold passes. After the time thresholdpasses, system 100 avoids triggering restocking events. In particular,although system 100 determines that at t3 the quantity of the perishableproducts drops down and reaches the quantity threshold, it decides toforgo initiating a restocking event because t3 is later than the timethreshold.

The fourth time period is from t3 to t4. In the fourth time period, thequality of the perishable products drops down and reaches the qualitythreshold at t4. Similar to the situation in the third time period,system 100 decides to forgo initiating a restocking event because t4 islater than the time threshold. The fifth time period is from t4 to22:00. The quantity and the quality of the perishable products continuesdropping down, but system 100 still does not initiating a thirdrestocking event.

Consistent with some embodiments illustrated in graph 2200, system 100may be configured to avoid from triggering actions that will enhance thestate of certain types of perishable products (e.g., increase theirquantity or improve their quality) towards the end-of-day or towards theend-of-week. In one implementation, a processing device of system 100may analyze the received set of images to determine EP quantityindicators 1902 indicative of quantities of certain types of perishableproducts at different times of day. At a first time of day (e.g., t1)when a quantity of products from the certain type of perishable productsreaches or below a quantity threshold, the processing device mayinitiate an action for restocking the shelving unit with additionalproducts from the certain type of perishable products. At a second timeof day (e.g., t3), later than the first time of day, when a quantity ofproducts from the certain type of perishable products reaches below thequantity threshold, the processing device may decide to forgo initiatingthe action for restocking the shelving unit with additional productsfrom the certain type of perishable products. The decision to forgoinitiating the action for restocking may be based on the time and date,based on inventory level, or based on store's policy. In some cases,processing device of system 100 may predetermine a value to the quantitythreshold (or the quality threshold) specifically for the certain typeof perishable products based on EP and SP indicators associated with thecertain type of perishable products. For example, the value of thethreshold may be determined based on the estimated time of arrival ofthe next shipment of products from the certain type.

In another implementation, a processing device of system 100 may analyzethe set of images to determine EP quality indicators 1900 indicative ofqualities of certain types of perishable products at different times ofday. At a first time of day (e.g., t2) when a quality of products fromthe certain type of perishable products is below a quality threshold,the processing device may initiate an action for improving the qualityassociated with the products of the certain type of perishable products.In one example, the action for improving the quality may includeremoving rotten products from the pile and adding additional products tothe pile. At a second time of day (e.g., t4), later than the first timeof day, when a quality of products of the certain type of perishableproducts is below the quantity threshold, a processing device of system100 may forgo initiating the action for improving the quality associatedwith the products of the certain type of perishable products.

Further to the discussion above about determining EP quality indicator1900, in one embodiment, the processing device of system 100 maydetermine that the quality associated with the products of the certaintype of perishable products is below a quality threshold determined forthe certain type of perishable products. For example, by counting anumber of times that customers engaged with a product of the certaintype of perishable products but did not retain the product of thecertain type of perishable products. The processing device of system 100may count the number of times based on analysis of the received set ofimages. The underlying assumption of this embodiment is that ifcustomers that were interested in this type of perishable productsdecided not to buy them, their quality may be not high enough. Inanother embodiment, the processing device of system 100 may determinethat the quality associated with the products of the certain type ofperishable products is below the quality threshold by comparing qualityreference data with visual characteristics of the products displayed onthe shelving unit, the visual characteristics being determined byanalyzing the set of images. For example, the color distribution of apile of bananas (e.g., green, yellow, dark yellow) may be indicative tothe condition of the bananas. In another embodiment, the processingdevice of system 100 may determine that the quality associated with theproducts of the certain type of perishable products is below the qualitythreshold by comparing quality reference data with an estimated timeduration that the products have been displayed on the shelving unit. Thequality reference data may be specific to the certain type of perishableproducts and specific to the time of the year. For example, when peachesstay on the shelf for more than two days, they are considered to be oflow quality. In contrast, avocados can stay on the shelf for seven daysbefore they are considered to be of low quality.

As mentioned above, the processing device of system 100 may initiate anaction for improving the quality associated with the products of acertain type of perishable products displayed on a shelving unit ofretail store 105. The action for improving the quality associated withthe products of the certain type of perishable products may improve anaverage quality, improve a quality of a selected group of instances ofthe perishable products (such as of 90% of the instances with thehighest/lowest quality), improve the median quality, improve the modequality, etc. Consistent with the present disclosure, the processingdevice of system 100 may analyze a set of images received from imagecapturing devices 125 to predict a time when the quality associated withthe products of the certain type of perishable products will be belowthe quantity threshold. Thereafter, the processing device of system 100may schedule a task for improving the quality associated with theproducts of the certain type of perishable products before the predictedtime. The task may be implemented by a store employee (human or robot).In one example, the action for improving the quality associated with theproducts of the certain type of perishable products may includetransmitting instructions for adding to the shelving unit products ofthe certain type of perishable products currently available in a storagearea associated with the retail store. In another example, the actionfor improving the quality associated with the products of the certaintype of perishable products may include transmitting instructions forsorting out low-quality products being displayed on the shelving unit.

In one specific embodiment, the at least one processing device of system100 may analyze a set of images to determine whether a quantity of theproducts of a certain type of perishable products is below a quantitythreshold. In response to a determination that the quantity of theproducts of the certain type of perishable products is below (orpredicted to reach) the quantity threshold, the at least one processingdevice may initiate an action for restocking the shelving unit withadditional products from the certain type of perishable products. Thedecision to restock the shelving unit with additional products from thecertain type of perishable products occurs when the specific time is,for example, about t=t1 in graph 2200. In addition, in response to adetermination that the quantity of the products of the certain type ofperishable products is above the quantity threshold, the at least oneprocessing device may forgo initiating the action for restocking theshelving unit with additional products from the certain type ofperishable products. The decision to forgo initiating the action forrestocking the shelving unit with additional products occurs when thespecific time is, for example, between t1 and t2 in graph 2200. The atleast one processing device of system 100 may also analyze the set ofimages to determine that the quantity of the products of the certaintype of perishable products is above the quantity threshold but aquality associated with the products of the certain type of perishableproducts is below a quality threshold. In response to a determinationthat the quantity of the products of the certain type of perishableproducts is above the quantity threshold but the quality associated withthe products of the certain type of perishable products is below thequality threshold, the at least one processing device may initiate anaction for improving the quality associated with the products of thecertain type of perishable products. The decision to improve the qualityof perishable products while the quantity of perishable products isabove the quantity threshold occurs when the specific time is, forexample, about t=t2 in graph 2200.

FIG. 23 depicts a flowchart of an example process 2300 executed by aprocessing device of system 100 (e.g., processing device 202) formanaging the quality and quantity of perishable products in retail store105. For purposes of illustration, in the following description,reference is made to certain components of system 100. It will beappreciated, however, that other implementations are possible and thatother components may be utilized to implement example process 2300. Itwill also be readily appreciated that the example process 2300 can bealtered to modify the order of steps, delete steps, or further includeadditional steps.

Process 2300 begins when the processing device analyzes a received setof images to determine a quantity of the perishable products from acertain type of perishable products (block 2302), for example, asdescribed above. The received set of images depicts the state of theperishable products from the certain type at a specific time.Thereafter, the processing device determines if the quantity of theperishable products from the certain type is greater than a quantitythreshold (decision block 2304). In the example illustrated in FIG. 22,the quantity threshold was determined to be 50% for that certain type ofperishable products. When the quantity of the products is below thequantity threshold, the processing device may determine if the specifictime is greater than the time threshold (decision block 2306). In theexample illustrated in FIG. 22, the time threshold was set to 6 PM. Inother examples, the time threshold may be set to a selected time in day,to a selected duration before planned closure of the retail store, andso forth. When the specific time is before the time threshold, theprocessing device may review inventory records to determine availabilityof the products of the certain type of perishable products in thestore's storage area (block 2308). Thereafter, the processing device maydetermine if additional products of the certain type of perishableproducts are currently available in a storage area associated with theretail store (decision block 2310). When additional products of thecertain type of perishable products are currently available in thestorage area, the processing device may initiate an action forrestocking a shelving unit associated with the certain type ofperishable products (block 2312). However, when additional products ofthe certain type of perishable products are not available in the storagearea, and/or when the specific time is later than the time threshold,the processing device may determine to forgo initiating an action forrestocking a shelving unit associated with the certain type ofperishable products (block 2314). In some examples, the quantitythreshold of decision block 2304 may be selected based on the type ofthe perishable product (for example, selecting a first quantitythreshold for a first type of perishable product, and selecting a secondquantity threshold for a second type of perishable product, where thefirst quantity threshold may differ from the second quantity threshold),the current time in day (for example, selecting a first quantitythreshold at a first time in day, and selecting a second quantitythreshold at a second time in day, where the first quantity thresholdmay differ from the second quantity threshold), the current day in week(for example, selecting a first quantity threshold at a first day inweek, and selecting a second quantity threshold at a second day in week,where the first quantity threshold may differ from the second quantitythreshold), the current date (for example, selecting a first quantitythreshold at a first date, and selecting a second quantity threshold ata second date, where the first quantity threshold may differ from thesecond quantity threshold, for example when the first date correspond toa holiday and the second date do not correspond to a holiday), thelocation of the perishable product within the retail store (for example,selecting a first quantity threshold at a first location, and selectinga second quantity threshold at a second location, where the firstquantity threshold may differ from the second quantity threshold), onthe type of the retail store (for example, selecting a first quantitythreshold at a first type of retail store, and selecting a secondquantity threshold at a second type of retail store, where the firstquantity threshold may differ from the second quantity threshold), andso forth.

The processing device may also analyze the received set of images todetermine the quality of the perishable products of the certain type(block 2315), for example, as described above. Thereafter, theprocessing device determines if the quality of the products of thecertain type is greater than a quality threshold (decision block 2316).In the example illustrated in FIG. 22, the quality threshold wasdetermined to be 75% for the certain type of perishable products. Whenthe quality of the products is below the quality threshold, theprocessing device may determine if the specific time is greater than thetime threshold (decision block 2318). In some examples, the timethreshold may be set to a selected time in day, to a selected durationbefore planned closure of the retail store, and so forth. The timethreshold of decision block 2318 may be identical to or different fromthe time threshold of decision block 2306. When the specific time isbefore than the time threshold, the processing device may reviewinventory records (block 2320). Thereafter, the processing device maydetermine if additional products of the certain type of perishableproducts are currently available in a storage area associated with theretail store (decision block 2322). When additional products of thecertain type of perishable products are currently available in thestorage area, the processing device may initiate an action for improvingthe quality of the perishable products of the certain type (block 2324).However, when the quality of the perishable products are greater thanthe threshold, when additional products of the certain type ofperishable products are not available in the storage area, and/or whenthe specific time is after than the time threshold, the processingdevice may forgo initiating an action for improving the quality of theperishable products of the certain type (block 2326). In one specificcase, the action for improving the quality of the perishable products ofthe certain type may include removing rotten and/or expired products anddoes not include restocking the shelf with additional products. In thiscase, the action for improving the quality of the perishable products ofthe certain type may occur even when there are currently available in astorage area associated with the retail store (decision block 2322). Insome examples, the quality threshold may be selected based on the typeof the perishable product (for example, selecting a first qualitythreshold for a first type of perishable product, and selecting a secondquality threshold for a second type of perishable product, where thefirst quality threshold may differ from the second quality threshold),the current time in day (for example, selecting a first qualitythreshold at a first time in day, and selecting a second qualitythreshold at a second time in day, where the first quality threshold maydiffer from the second quality threshold), the current day in week (forexample, selecting a first quality threshold at a first day in week, andselecting a second quality threshold at a second day in week, where thefirst quality threshold may differ from the second quality threshold),the current date (for example, selecting a first quality threshold at afirst date, and selecting a second quality threshold at a second date,where the first quality threshold may differ from the second qualitythreshold, for example when the first date correspond to a holiday andthe second date do not correspond to a holiday), the location of theperishable product within the retail store (for example, selecting afirst quality threshold at a first location, and selecting a secondquality threshold at a second location, where the first qualitythreshold may differ from the second quality threshold), on the type ofthe retail store (for example, selecting a first quality threshold at afirst type of retail store, and selecting a second quality threshold ata second type of retail store, where the first quality threshold maydiffer from the second quality threshold), and so forth.

Consistent with the present disclosure, the processing device action forrestocking the shelving unit with additional products in block 2312 mayinclude transmitting instructions for adding to the shelving unitproducts of the certain type of perishable products currently availablein a storage area associated with retail store 105. In one embodiment,after determining that an action prompting restocking the shelving unitis needed, the at least one processing device may determine that thereare no available products of the certain type of perishable productsavailable in the storage area associated with the retail store (e.g.,decision block 2310 and 2322). In response to the determination thatthere are no available products of the certain type of perishableproducts in the storage area associated with retail store 105, the atleast one processing device may withhold from initiating the actionprompting restocking the shelving unit. In another embodiment, inresponse to the determination that that there are no available productsof the certain type of perishable products in the storage areaassociated with retail store 105, the at least one processing device maysuggest placing on the shelving unit products from a different type ofperishable product that are currently available in the storage areaassociated with retail store 105.

Consisted with an embodiment of the present disclosure, a processingdevice of system 100 may use artificial neural networks and machinelearning algorithms to identify possible relationships between two ormore types of products. The identified relationships may be forperishable products with regular retail products, or for perishableproducts with another type of perishable products. In one embodiment,the processing device may use the identified relationships to determineEP and SP suggestions, such as EP location suggestion 1922 and SPlocation suggestion 1928. For example, the at least one processingdevice may recommend placing tortilla chips and a recipe for guacamolenext to ripe avocados. In another embodiment, the processing device mayuse the identified relationships to determine EP suggestions for a firsttype of perishable products at least partially based on indicatorsassociated with a second type of perishable products.

FIG. 24 illustrates a shelving unit 2400 that includes a first displayarea 2402 for a first type of perishable products (e.g., pears) and asecond display area 2404 for a second type of perishable productsdifferent from the first type of perishable product (e.g., apples).Consistent with the present disclosure, shelving unit 2400 may bemonitored by a plurality of image capturing devices 125 (e.g., a firstcamera 2406 may be positioned above shelving unit 2400 and a secondcamera 2408 may be positioned to the side of shelving unit 2400). In oneembodiment, an optical axis of first camera 2406 may be substantiallyvertical. For example, between −10° and 10° relative to a normal of thefloor of retail store 105.

Consistent with the present disclosure, at least one processing deviceof system 100 may receive a set of images depicting a plurality ofperishable products displayed on shelving unit 2400. In one embodiment,the set of images may be captured using at least one image capturingdevice 125 (such as camera 2406 and/or camera 2408). The set of imagesmay be analyzed to identify a first area of shelving unit 2400 thatincludes perishable products and a second area of shelving unit 2400that does not include perishable products. At least one processingdevice of system 100 may analyze the set of images to determine one ormore EP indicators for perishable products from the first type ofperishable products found on first display area 2402 and one or more EPindicators for perishable products from the second type of perishableproducts found on second display area 2404. In one example, the set ofimages may be analyzed to determine information to a user, such as aplanogram compliance report, a shelf utilization report, and so forth.

In an embodiment, at least one processing device of system 100 mayidentify in the set of images representations of one or more products ofa first type of perishable of products (e.g., pears) and one or moreproducts of a second type of perishable products different from thefirst type of perishable products (e.g., apples). The at least oneprocessing device of system 100 may analyze the set of images todetermine information about a displayed inventory of the first type ofperishable products and to determine information about a displayedinventory of the second type of perishable products. The determinedinformation about the displayed inventories of the first and secondtypes of perishable products may include one or more of: EP qualityindicator 1900, EP quantity indicator 1902, EP demand indicator 1904, EPprice indicator 1906, and EP planogram indicator 1908. In addition, theat least one processing device of system 100 may obtain informationabout additional perishable products of the first and second types ofperishable products scheduled to be delivered to retail store 105. Theobtained information about the scheduled products of the first andsecond types of perishable products may include SP quality indicator1910, SP quantity indicator 1912, SP demand indicator 1914, SP costindicator 1916, SP shelf-life indicator 1918, and SP ETA indicator 1920.

The at least one processing device of system 100 may determine toinitiate an action prompting a clearance sale of the products of thefirst type of perishable products based on the determined informationabout the displayed inventory of the first type of perishable product,the obtained information about additional perishable products of thefirst type of perishable products scheduled to be delivered to retailstore 105, or a combination of both. In a similar manner, the at leastone processing device of system 100 may determine one or more indicatorsfor the second type of perishable products. Thereafter, the at least oneprocessing device of system 100 may determine to avoid initiating anaction prompting a clearance sale of products of the second type ofperishable products based on: the determined information about thedisplayed inventory of the first type of perishable product, theobtained information about additional perishable products of the firsttype of perishable products scheduled to be delivered to retail store105, the determined information about the displayed inventory of thesecond type of perishable products, the obtained information aboutadditional perishable products of the second type of perishable productsscheduled to be delivered to retail store 105, or a combination of them.

FIG. 25 illustrates three bar diagrams showing three scenarios in whichthe prices of the first and second types of products change with respectto changes of quality over time. In the three bar diagrams in FIG. 25the quality of perishable products from the first and second types ofproducts is depicted as the only factor for determining the price of thefirst and second types of products. However, one of ordinary skill inthe art will recognize that any EP indicator or any SP indicator (or anycombination of indicators) associated with perishable products from thefirst and second types of products may be used in determining the priceof the first and second types of products. For example, the processingdevice may initiate an action prompting a clearance sale of perishableproducts from a first type based on data about the upcoming shipment ofperishable products of the second type of products.

In a first bar diagram 2500, the quality of the apples declines overtime and the quality of the pears remains substantially the same. Asillustrated in first bar diagram 2500, at t=t3 the processing devicedetermines to reduce the price of the apples. The price of the apples isbeing differentially reduced throughout a period of time from $4 per Kgto $2.75 per Kg and the price of the pears remains $5 per Kg. In asecond bar diagram 2502, the quality of the pears declines over time andthe quality of the apples remains substantially the same. As illustratedin second bar diagram 2502, at t=t3 the processing device may determinereduce the price of the pears. The price of the pears is beingdifferentially reduced throughout a period of time from $5 per Kg to $4per Kg and the price of the apples remains $4 per Kg. In a third bardiagram 2504, both the quality of the apples and the quality of thepears decline over time. As illustrated in third bar diagram 2504, att=t3 the processing device may determine to differentially reduce theprice of the apples while determining to keep the price of the pears at$5 per Kg. The price of the pears remains $5 per Kg even when theirquality declines.

The following describes an example scenario that would explain why inthird bar diagram 2504 the processing device determines todifferentially reduce the price of the apples from $4 to $2.75 whilekeeping the price of the pears at $5 per Kg. In the example scenario,the demand for pears is higher than the demand for apples, and retailstore 105 has a rule that when the quality of a product is below 30%, itshould be thrown away. Consistent with third bar diagram 2504, at t=t3,the processing device may predict that the quality of the apples willdecline faster than the quality of the pears. Retail store 105 hasincentive to sell as many apples before their quality reaches to the 30%threshold. Based on the determined relationship between apples andpears, the processing device predicts that lowering the price of thepears will hurt the sale of the apples. Therefore, the processing devicemay avoid initiating an action prompting a clearance sale of pears.

In some embodiments, the processing device may determine if initiatingthe action prompting a clearance sale of the products of the first typeof perishable products is needed based on information about products ofthe first type of perishable products available in a storage areaassociated with retail store 105. In addition, the processing device maydetermine if initiating the action prompting a clearance sale of theproducts of the first type of perishable products is needed based oninformation about products of the first type of perishable productsscheduled to be delivered to retail store 105. In a similar example,assuming the demand for pears is higher than the demand for apples and ashipment of apples is scheduled to arrive soon, retail store 105 mayhave an incentive to sell as many apples before the shipment of applesarrives. Based on the determined relationship between apples and pears,the processing device may predict that lowering the price of the pearswill hurt the sale of the apples. Therefore, the processing device maynote initiate an action prompting a clearance sale of pears.

In other embodiments, the processing device may use EP indicatorsassociated with products of the first type of perishable products todetermine whether the products from the first type of perishableproducts need to be cleared by the end of the day. Alternatively, theprocessing device may use EP indicators associated with products of thefirst type of perishable products to determine whether the products fromthe first type of perishable products need to be cleared by the end ofthe week. In case the processing device determines that the products ofthe first type of perishable products need to be cleared by the end of agiven time window, it may initiate the action prompting the clearancesale of the products of the first type of perishable products. Forexample, as discussed above, the processing device may use one or moreEP indicators to predict that the condition of the products of the firsttype of perishable products will be below a quality threshold by the endof the day (or the end of the week) and accordingly initiate the actionprompting the clearance sale.

Consistent with the present disclosure, the action prompting a clearancesale of the products of the first type of perishable products mayinclude transmitting instructions to differentially lower a price of thefirst type of perishable products throughout a period of time. Theprocessing device may determine a rate for lowering the price of thefirst type of perishable products based on changes in the determinedquality of products of the first type. For example, when the first typeof perishable products should be cleared by the end of the day, theprices may change every hour. Alternatively, the action prompting aclearance sale of the products of the first type of perishable productsincludes transmitting instructions to move the products of the firsttype of perishable products to a location with a higher customerthroughput. With reference to the example discussed above, the applesmay be moved to first display area 2402, and the pears may be moved tosecond display area 2404. In specific cases where the first type ofperishable products is being produced in retail store 105 (e.g., bakedgoods), the action prompting a clearance sale may include transmittinginstructions to stop producing more products of the first type ofperishable products.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray,or other optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

What is claimed is:
 1. A system for identifying perishable products in aretail store based on analysis of image data and for automaticallygenerating suggestions relating to the identified products, the systemcomprising: at least one processor configured to: receive, from an imagecapture device, at least one hyperspectral digital image depicting aplurality of perishable products displayed on at least one shelving unitin a retail store; transform the at least one hyperspectral digitalimage to generate transformation digital image data by using atransformation function, the transformation function comprising at leastone of a convolution, a visual filter, or a nonlinear function; analyzethe transformation digital image data using an image analysis techniqueto determine information about a displayed inventory of the plurality ofperishable products, the information comprising an indicator of anexisting product, the analysis of the transformation image datacomprising: comparing the transformation digital image data to a productquality profile pattern, the product quality profile pattern having beengenerated by a first machine learning model, the first machine learningmodel having been trained using combinations of digital hyperspectralimages and corresponding levels of product qualities including measuredlevels of hyperspectral wavelength information or waveband informationfor chemical constituents; and determining that values of thetransformation digital image data match values of the product qualityprofile pattern within a value threshold; receive information from atleast one external source, the information indicating additionalperishable products scheduled to be displayed on the at least oneshelving unit; analyze, using a second machine learning model, thereceived information to determine an indicator of a scheduled productassociated with the additional perishable products, wherein the secondmachine learning model is configured to learn relationships betweenproduct scheduling information and a state of products when delivered topredict a future state of products after delivery; generate, based onthe indicator of the existing product and the indicator of the scheduledproduct, at least one suggestion regarding placement of perishableproducts in the retail store; and transmit, to a device associated withan entity of the retail store, instructions for implementing the atleast one suggestion.
 2. The system of claim 1, wherein the informationabout the displayed inventory includes quantity indicators for multipletypes of perishable products, and wherein the at least one processor isconfigured to: determine the quantity indicators for the multiple typesof perishable products based on analysis of the at least one image; anduse the determined quantity indicators for the multiple types ofperishable products to generate the at least one suggestion regardingplacement of perishable products in the retail store.
 3. The system ofclaim 1, wherein the information about the displayed inventory includesquality indicators for multiple types of perishable products, andwherein the at least one processor is configured to: determine thequality indicators for the multiple types of perishable products basedon analysis of the at least one image; and use the determined qualityindicators for the multiple types of perishable products to generate theat least one suggestion regarding placement of perishable products inthe retail store.
 4. The system of claim 1, wherein the informationabout the displayed inventory includes demand indicators for multipletypes of perishable products, and wherein the at least one processor isconfigured to: determine the demand indicators for the multiple types ofperishable products based on analysis of the at least one image; and usethe determined demand indicators for the multiple types of perishableproducts to generate the at least one suggestion regarding placement ofperishable products in the retail store.
 5. The system of claim 1,wherein the received information indicating the additional perishableproducts scheduled to be displayed on the at least one shelving unitincludes a quantity indicator for the additional perishable products,and wherein the at least one processor is configured to generate the atleast one suggestion regarding placement of the additional perishableproducts based on the quantity indicator of the additional perishableproducts.
 6. The system of claim 1, wherein the received informationindicating the additional perishable products scheduled to be displayedon the at least one shelving unit includes a quality indicator for theadditional perishable products, and wherein the at least one processoris configured to generate the at least one suggestion regardingplacement of the additional perishable products based on the qualityindicator for the additional perishable products.
 7. The system of claim1, wherein the received information indicating the additional perishableproducts scheduled to be displayed on the at least one shelving unitincludes a shelf-life estimation of the additional perishable products,and wherein the at least one processor is configured to generate the atleast one suggestion regarding placement of the additional perishableproducts based on the shelf-life estimation of the additional perishableproducts.
 8. The system of claim 1, wherein the received informationindicating the additional perishable products scheduled to be displayedon the at least one shelving unit includes costs associated with theadditional perishable products, and wherein the at least one processoris configured to generate the at least one suggestion regardingplacement of the additional perishable products based on the costs ofthe additional perishable products.
 9. The system of claim 1, whereinthe received information indicating the additional perishable productsscheduled to be displayed on the at least one shelving unit includesestimated arrival time of the additional perishable products, andwherein the at least one processor is configured to generate the atleast one suggestion regarding placement of the additional perishableproducts based on the estimated arrival time of the additionalperishable products.
 10. The system of claim 1, wherein the receivedinformation indicating the additional perishable products scheduled tobe displayed on the at least one shelving unit includes a predicteddemand for the additional perishable products, and wherein the at leastone processor is configured to generate the at least one suggestionregarding placement of the additional perishable products based on thepredicted demand for the additional perishable products.
 11. The systemof claim 1, wherein the at least one suggestion includes an indicationof a temporary location for at least some of the plurality of perishableproducts to reside prior to arrival of the additional perishableproducts, and wherein the at least one processor is configured todetermine the temporary location for the at least some of the pluralityof perishable products based on the information about the displayedinventory and the information indicating the additional perishableproducts.
 12. The system of claim 1, wherein the at least one placementsuggestion includes an indication of a temporary size of a display areafor at least some of the plurality of perishable products to be usedprior to arrival of the additional perishable products, and wherein theat least one processor is configured to determine the temporary sizebased on the information about the displayed inventory and theinformation indicating the additional perishable products.
 13. Thesystem of claim 1, wherein the at least one placement suggestionincludes an indication of a temporary price for at least some of theplurality of perishable products to be applied prior to arrival of theadditional perishable products, and wherein the at least one processoris configured to determine the temporary price based on the informationabout the displayed inventory and the information indicating theadditional perishable products.
 14. The system of claim 1, wherein theat least one placement suggestion includes an indication of a locationfor placing the additional perishable products on the at least oneshelving unit, and wherein the at least one processor is configured todetermine the location for placing the additional perishable productsbased on the information about the displayed inventory and theinformation indicating the additional perishable products.
 15. Thesystem of claim 1, wherein the at least one placement suggestionincludes an indication of a size for a display area for the additionalperishable products, and wherein the at least one processor isconfigured to determine the size for the display area based on theinformation about the displayed inventory and the information indicatingthe additional perishable products.
 16. The system of claim 1, whereinthe at least one placement suggestion includes a price for theadditional perishable products, and wherein the at least one processoris configured to determine the price for the additional perishableproducts based on the information about the displayed inventory and theinformation indicating the additional perishable products.
 17. Thesystem of claim 1, wherein the at least one processor is furtherconfigured to: receive information about perishable products availablein a storage area of the retail store; and use the information about theperishable products available in the storage area and the informationindicating the additional perishable products to determine at least onetask for reorganizing perishable products in the retail store.
 18. Thesystem of claim 17, wherein the at least one task for reorganizingperishable products includes a task for reorganizing perishable productscurrently available in the storage area.
 19. The system of claim 17,wherein the at least one task for reorganizing perishable productsincludes a task for reorganizing perishable products placed on the atleast one shelving unit.
 20. The system of claim 17, wherein the atleast one processor is further configured to: predict a condition of theperishable products available in the storage area of the retail store ata time when the additional perishable products are scheduled to arrive;and use the information about the perishable products available in thestorage area, the information indicating the additional perishableproducts, and the predicted condition to determine at least one task forreorganizing perishable products in the retail store.
 21. The system ofclaim 1, wherein the at least one processor is further configured toanalyze the transformation image data by distinguishing a first productrepresented in the transformation image data from a second product usinga classification algorithm generated by a third model learning model.22. The system of claim 1, wherein comparing the transformation imagedata to the product quality profile pattern is performed as part of asequence of fuzzy matching between the transformation image data and aplurality of product quantity profile patterns stored in a database. 23.The system of claim 1, wherein the at least one external source is asensor in a product transportation vehicle.
 24. The system of claim 1,wherein the generated suggestion is dependent on at least one of: aproduct quantity, a type of the plurality of perishable products, acurrent date, a current time, a product location within the retailstore, or a type of retail store.
 25. The system of claim 1, wherein thefirst and second machine learning models are each at least one of: adeep neural network or a convolutional neural network.
 26. The system ofclaim 1, wherein the second machine learning model is a regression modeland analyzing the received information to determine an indicator of ascheduled product comprises predicting a state of the additionalperishable products at a scheduled delivery time.
 27. The system ofclaim 1, wherein the information about the displayed inventory of theplurality of perishable products is further determined based on analysisof data captured by at least one weight sensor or at least one pressuresensor.
 28. The system of claim 1, wherein the analysis of thetransformation image data further comprises determining a ripenessassociated with the plurality of perishable products based on thedetermining that the values of the transformation digital image datamatch the values of the product quality profile pattern within the valuethreshold.
 29. The system of claim 1, wherein transforming the at leastone hyperspectral digital image comprises constructing a 3D image.
 30. Amethod for identifying perishable products in a retail store based onanalysis of image data and for automatically generating suggestionsrelating to the identified products, the method comprising: receiving,from an image capture device, at least one hyperspectral digital imagedepicting a plurality of perishable products displayed on at least oneshelving unit in a retail store; transform the at least onehyperspectral digital image to generate transformation digital imagedata by using a transformation function, the transformation functioncomprising at least one of a convolution, a visual filter, or anonlinear function; analyzing the transformation digital image datausing an image analysis technique to determine information about adisplayed inventory of the plurality of perishable products, theinformation comprising an indicator of an existing product, the analysisof the transformation image data comprising: comparing thetransformation digital image data to a product quality profile pattern,the product quality profile pattern having been generated by a firstmachine learning model, the first machine learning model having beentrained using combinations of hyperspectral digital images andcorresponding levels of product qualities including measured levels ofhyperspectral wavelength information or waveband information forchemical constituents; and determining that values of the transformationdigital image data match values of the product quality profile patternwithin a value threshold; receiving information from at least oneexternal source, the information indicating additional perishableproducts scheduled to be displayed on the at least one shelving unit;analyzing, using a second machine learning model, the receivedinformation to determine an indicator of a scheduled product associatedwith the additional perishable products, wherein the second machinelearning model is configured to learn relationships between productscheduling information and a state of products when delivered to predicta future state of products after delivery; generating, based on theindicator of the existing product and the indicator of the scheduledproduct, at least one suggestion regarding placement of perishableproducts in the retail store; and transmitting, to a device associatedwith an entity of a retail store, instructions for implementing the atleast one suggestion.
 31. A computer program product for identifyingperishable products in a retail store based on analysis of image dataand for automatically generating placement suggestions relating to theidentified products embodied in a non-transitory computer-readablemedium and executable by at least one processor, the computer programproduct including instructions for causing the at least one processor toexecute a method comprising: receiving, from an image capture device, atleast one hyperspectral digital image depicting a plurality ofperishable products displayed on at least one shelving unit in a retailstore; transform the at least one hyperspectral digital image togenerate transformation digital image data by using a transformationfunction, the transformation function comprising at least one of aconvolution, a visual filter, or a nonlinear function; analyzing thetransformation digital image data using an image analysis technique todetermine information about a displayed inventory of the plurality ofperishable products, the information comprising an indicator of anexisting product, the analysis of the transformation image datacomprising: comparing the transformation digital image data to a productquality profile pattern, the product quality profile pattern having beengenerated by a first machine learning model, the first machine learningmodel having been trained using combinations of hyperspectral digitalimages and corresponding levels of product qualities including measuredlevels of hyperspectral wavelength information or waveband informationfor chemical constituents; and determining that values of thetransformation digital image data match values of the product qualityprofile pattern within a value threshold; receiving information from atleast one external source, the information indicating additionalperishable products scheduled to be displayed on the at least oneshelving unit; analyzing, using a second machine learning model, theinformation to determine an indicator associated with a scheduledproduct, wherein the second machine learning model is configured tolearn relationships between product scheduling information and a stateof products when delivered to predict a future state of products afterdelivery; generating, based on the indicator of the existing product andthe indicator of the scheduled product, at least one suggestionregarding placement of perishable products in the retail store; andtransmitting, to a device associated with an entity of the retail store,instructions for implementing providing the at least one suggestion.